Jul 17 21:25:32.608033 ip-10-0-139-178 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 21:25:32.608041 ip-10-0-139-178 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 21:25:32.608048 ip-10-0-139-178 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 21:25:32.608273 ip-10-0-139-178 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 21:25:42.787729 ip-10-0-139-178 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 21:25:42.787745 ip-10-0-139-178 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot bb3e88139fa949edaaf01c69abcd3196 -- Jul 17 21:27:41.162882 ip-10-0-139-178 systemd[1]: Starting Kubernetes Kubelet... Jul 17 21:27:41.547339 ip-10-0-139-178 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:27:41.547339 ip-10-0-139-178 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 21:27:41.547339 ip-10-0-139-178 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:27:41.547339 ip-10-0-139-178 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 21:27:41.547339 ip-10-0-139-178 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:27:41.548025 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.547398 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 21:27:41.552386 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552368 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:41.552386 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552384 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:41.552386 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552388 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552391 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552394 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552397 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552401 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552403 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552406 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552409 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552411 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552414 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552416 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552419 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552421 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552424 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552427 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552429 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552432 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552442 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552445 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552448 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:41.552486 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552451 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552454 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552456 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552458 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552461 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552464 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552466 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552469 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552471 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552474 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552476 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552492 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552495 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552498 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552500 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552503 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552506 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552510 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552514 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552516 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:41.553000 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552519 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552521 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552523 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552526 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552528 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552531 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552534 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552536 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552538 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552542 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552546 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552548 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552551 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552554 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552557 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552560 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552563 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552567 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552570 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:41.553483 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552573 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552576 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552578 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552581 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552584 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552587 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552590 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552593 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552596 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552615 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552619 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552621 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552624 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552626 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552629 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552631 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552634 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552636 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552639 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552641 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:41.553955 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552643 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552646 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552648 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552651 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552653 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552655 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552658 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552661 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552664 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552666 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552669 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552671 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552674 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552677 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552679 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552681 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552684 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.552686 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553377 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:41.554422 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553383 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553386 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553388 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553391 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553395 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553398 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553400 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553403 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553405 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553408 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553410 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553413 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553415 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553418 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553420 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553423 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553425 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553427 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553430 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:41.554886 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553433 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553436 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553439 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553442 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553444 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553446 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553449 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553451 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553454 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553456 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553458 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553461 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553463 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553467 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553470 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553473 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553475 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553478 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553481 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553484 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:41.555359 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553486 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553489 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553492 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553495 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553498 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553500 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553503 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553505 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553508 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553510 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553513 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553515 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553519 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553522 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553525 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553528 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553530 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553532 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553535 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:41.555875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553537 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553540 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553543 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553545 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553548 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553550 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553552 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553555 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553558 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553561 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553563 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553565 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553568 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553571 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553573 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553576 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553578 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553581 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553583 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553586 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:41.556331 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553588 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553591 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553593 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553595 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553616 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553619 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553622 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553625 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553627 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553630 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553633 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553635 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553638 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553641 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553645 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553648 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553651 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553654 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553657 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:41.556845 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.553659 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554019 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554025 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554028 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554031 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554042 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554045 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554048 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554050 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554053 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554056 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554058 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554061 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554063 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554066 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554068 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554070 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554073 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554075 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:41.557298 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554078 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554081 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554083 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554086 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554089 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554091 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554094 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554096 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554099 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554101 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554104 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554106 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554109 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554111 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554114 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554116 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554119 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554121 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554124 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554127 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:41.557776 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554129 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554131 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554134 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554138 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554141 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554144 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554146 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554150 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554153 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554156 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554159 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554162 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554165 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554167 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554170 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554172 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554175 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554177 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554180 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:41.558346 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554182 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554185 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554187 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554190 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554192 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554195 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554198 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554200 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554203 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554205 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554208 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554210 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554213 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554215 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554219 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554221 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554223 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554226 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554228 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554230 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:41.558820 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554233 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554235 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554237 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554240 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554243 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554245 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554248 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554251 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554253 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554256 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554258 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554260 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554263 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554265 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554268 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554270 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554272 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554274 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554277 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554279 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:41.559329 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554282 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554284 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554355 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554362 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554368 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554374 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554381 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554385 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554391 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554397 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554402 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554406 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554409 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554412 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554415 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554418 2566 flags.go:64] FLAG: --cgroup-root="" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554424 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554429 2566 flags.go:64] FLAG: --client-ca-file="" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554432 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554436 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554441 2566 flags.go:64] FLAG: --cluster-domain="" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554443 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554447 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554450 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 21:27:41.559831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554453 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554458 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554460 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554464 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554467 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554470 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554473 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554476 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554479 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554482 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554486 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554489 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554491 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554494 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554497 2566 flags.go:64] FLAG: --enable-server="true" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554501 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554505 2566 flags.go:64] FLAG: --event-burst="100" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554508 2566 flags.go:64] FLAG: --event-qps="50" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554510 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554513 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554516 2566 flags.go:64] FLAG: --eviction-hard="" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554520 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554522 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554525 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554528 2566 flags.go:64] FLAG: --eviction-soft="" Jul 17 21:27:41.560399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554533 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554535 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554538 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554541 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554544 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554546 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554549 2566 flags.go:64] FLAG: --feature-gates="" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554553 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554556 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554558 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554561 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554564 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554567 2566 flags.go:64] FLAG: --help="false" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554570 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-139-178.ec2.internal" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554573 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554576 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554579 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554582 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554585 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554588 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554590 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554593 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554608 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 21:27:41.561071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554612 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554615 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554618 2566 flags.go:64] FLAG: --kube-reserved="" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554621 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554623 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554626 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554629 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554632 2566 flags.go:64] FLAG: --lock-file="" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554635 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554639 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554642 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554647 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554650 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554653 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554656 2566 flags.go:64] FLAG: --logging-format="text" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554659 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554662 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554664 2566 flags.go:64] FLAG: --manifest-url="" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554667 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554674 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554677 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554681 2566 flags.go:64] FLAG: --max-pods="110" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554684 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554687 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554689 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 21:27:41.561659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554692 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554695 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554698 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554701 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554708 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554711 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554714 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554717 2566 flags.go:64] FLAG: --pod-cidr="" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554720 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554726 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554729 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554732 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554735 2566 flags.go:64] FLAG: --port="10250" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554738 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554741 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-081f9c9ae4f07d447" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554744 2566 flags.go:64] FLAG: --qos-reserved="" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554747 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554751 2566 flags.go:64] FLAG: --register-node="true" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554754 2566 flags.go:64] FLAG: --register-with-taints="" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554757 2566 flags.go:64] FLAG: --registry-burst="10" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554760 2566 flags.go:64] FLAG: --registry-qps="5" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554763 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554765 2566 flags.go:64] FLAG: --reserved-memory="" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554769 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554772 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 21:27:41.562263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554775 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554777 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554785 2566 flags.go:64] FLAG: --runonce="false" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554788 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554791 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554794 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554797 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554799 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554802 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554805 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554808 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554810 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554813 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554816 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554819 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554822 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554825 2566 flags.go:64] FLAG: --system-cgroups="" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554828 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554833 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554836 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554839 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554843 2566 flags.go:64] FLAG: --tls-min-version="" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554846 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554848 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554853 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 21:27:41.562915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554856 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554858 2566 flags.go:64] FLAG: --v="2" Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554862 2566 flags.go:64] FLAG: --version="false" Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554867 2566 flags.go:64] FLAG: --vmodule="" Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554870 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.554873 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554988 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554994 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.554997 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555000 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555003 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555006 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555009 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555011 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555014 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555016 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555019 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555021 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555024 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555028 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:41.563513 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555031 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555034 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555037 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555040 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555043 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555045 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555048 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555050 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555055 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555057 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555061 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555065 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555068 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555071 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555074 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555077 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555080 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555082 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555085 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:41.564031 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555087 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555090 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555092 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555103 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555105 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555108 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555110 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555113 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555115 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555117 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555120 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555122 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555125 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555127 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555129 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555132 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555135 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555137 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555140 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555143 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:41.564488 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555146 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555149 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555152 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555155 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555159 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555161 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555163 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555166 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555168 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555171 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555174 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555176 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555179 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555181 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555183 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555186 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555189 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555191 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555194 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555196 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:41.564970 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555199 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555201 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555203 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555206 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555208 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555211 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555213 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555216 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555218 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555220 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555223 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555226 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555229 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555232 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555235 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555237 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555241 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555243 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555246 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555248 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:41.565455 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555251 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555254 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555256 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555259 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555261 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.555263 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.555709 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.562116 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.562133 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.562147 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.562153 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562209 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562214 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562217 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562220 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:41.565999 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562223 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562226 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562229 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562232 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562234 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562237 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562240 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562242 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562245 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562247 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562250 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562253 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562255 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562257 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562260 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562263 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562266 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562269 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562271 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562274 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:41.566379 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562277 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562279 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562282 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562284 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562286 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562291 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562293 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562296 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562299 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562301 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562303 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562306 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562309 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562311 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562316 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562319 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562321 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562324 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:41.566875 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562326 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562330 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562334 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562336 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562339 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562342 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562344 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562348 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562351 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562353 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562356 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562359 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562362 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562364 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562367 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562369 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562372 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562375 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562378 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562380 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:41.567347 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562383 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562386 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562390 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562393 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562396 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562399 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562402 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562404 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562406 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562409 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562411 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562413 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562416 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562418 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562421 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562423 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562426 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562428 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562431 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:41.567870 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562433 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562436 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562438 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562441 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562443 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562446 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562448 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562451 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562453 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562455 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562458 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562460 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562462 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562465 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562468 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562470 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562472 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:41.568320 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562475 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.562480 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562645 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562652 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562655 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562658 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562661 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562664 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562666 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562669 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562671 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562673 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562676 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562679 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562681 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:27:41.568859 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562684 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562687 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562689 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562691 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562694 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562698 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562702 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562704 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562707 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562710 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562712 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562715 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562717 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562720 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562722 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562725 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562727 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562730 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562732 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562735 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:27:41.569251 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562737 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562740 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562742 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562744 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562747 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562749 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562753 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562756 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562759 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562762 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562764 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562767 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562769 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562772 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562774 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562777 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562780 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562782 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562784 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562787 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:27:41.569765 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562789 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562792 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562794 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562797 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562799 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562802 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562804 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562806 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562810 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562812 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562814 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562817 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562819 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562821 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562824 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562826 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562828 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562831 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562833 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562835 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:27:41.570270 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562838 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562840 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562842 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562844 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562847 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562850 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562852 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562854 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562857 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562860 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562862 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562865 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562867 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562869 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562871 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562874 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562876 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562880 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562883 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:27:41.570854 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562885 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562888 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562891 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562893 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562895 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562897 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:41.562900 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.562904 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.563030 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.565774 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.566987 2566 server.go:1019] "Starting client certificate rotation" Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.567091 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.567161 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:27:41.571544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.569101 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:27:41.572857 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.572832 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:27:41.582826 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.582804 2566 log.go:25] "Validated CRI v1 runtime API" Jul 17 21:27:41.584722 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.584707 2566 log.go:25] "Validated CRI v1 image API" Jul 17 21:27:41.585594 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.585575 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 21:27:41.592891 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.592869 2566 fs.go:135] Filesystem UUIDs: map[28d43cd1-9d44-44ec-b6b1-6b6aa90bf5b7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fa40d785-529c-45f1-9e11-a34f9757bf26:/dev/nvme0n1p4] Jul 17 21:27:41.592955 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.592891 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 21:27:41.594670 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.594650 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:27:41.598229 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.598116 2566 manager.go:216] Machine: {Timestamp:2026-07-17 21:27:41.596856259 +0000 UTC m=+0.322482465 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098831 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a590d12991ba6106840bb2fa8447 SystemUUID:ec28a590-d129-91ba-6106-840bb2fa8447 BootID:bb3e8813-9fa9-49ed-aaf0-1c69abcd3196 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cb:0b:3c:27:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cb:0b:3c:27:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:0a:7d:bc:25:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 21:27:41.598229 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.598228 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 21:27:41.598344 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.598313 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 21:27:41.601303 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.601278 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 21:27:41.601449 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.601305 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-178.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 21:27:41.601932 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.601922 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 21:27:41.601969 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.601934 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 21:27:41.601969 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.601942 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:27:41.602915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.602904 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:27:41.604053 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.604041 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 21:27:41.606626 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.606594 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:27:41.606937 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.606927 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 21:27:41.606969 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.606963 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 17 21:27:41.606997 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.606976 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 21:27:41.606997 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.606987 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 21:27:41.606997 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.606996 2566 kubelet.go:398] "Adding apiserver pod source" Jul 17 21:27:41.607087 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.607005 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 21:27:41.607329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.607318 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:27:41.607371 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.607335 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:27:41.609882 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.609862 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 21:27:41.611362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.611345 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 21:27:41.611465 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.611385 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 21:27:41.612066 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612052 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 21:27:41.612141 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612073 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 21:27:41.612141 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612082 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 21:27:41.612141 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612090 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 21:27:41.612141 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612101 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 21:27:41.612141 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612110 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 21:27:41.612328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612204 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 21:27:41.612328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612214 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 21:27:41.612328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612225 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 21:27:41.612328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612234 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 21:27:41.612328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612247 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 21:27:41.612328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612260 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 21:27:41.612964 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612953 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 21:27:41.613030 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.612967 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 21:27:41.618143 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.618066 2566 server.go:1268] "Started kubelet" Jul 17 21:27:41.618283 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.618190 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 21:27:41.618505 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.618097 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 21:27:41.618569 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.618526 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 21:27:41.618634 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.618564 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 21:27:41.620847 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.620819 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 21:27:41.620982 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.620874 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nsxwl" Jul 17 21:27:41.621202 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.621179 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 21:27:41.621305 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.621281 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 21:27:41.621380 ip-10-0-139-178 systemd[1]: Started Kubernetes Kubelet. Jul 17 21:27:41.621876 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.621858 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 17 21:27:41.624151 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.624129 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 21:27:41.625263 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.624338 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-178.ec2.internal.18c330ea9f10f4ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-178.ec2.internal,UID:ip-10-0-139-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-178.ec2.internal,},FirstTimestamp:2026-07-17 21:27:41.618017486 +0000 UTC m=+0.343643673,LastTimestamp:2026-07-17 21:27:41.618017486 +0000 UTC m=+0.343643673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-178.ec2.internal,}" Jul 17 21:27:41.627848 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.627834 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nsxwl" Jul 17 21:27:41.630657 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.630641 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 21:27:41.630729 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.630657 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 21:27:41.630729 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.630715 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:41.631208 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631190 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 21:27:41.631208 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631192 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 21:27:41.631351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631243 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 21:27:41.631400 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631382 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 17 21:27:41.631400 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631391 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 17 21:27:41.631545 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631515 2566 factory.go:55] Registering systemd factory Jul 17 21:27:41.631545 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631537 2566 factory.go:223] Registration of the systemd container factory successfully Jul 17 21:27:41.631809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631776 2566 factory.go:153] Registering CRI-O factory Jul 17 21:27:41.631809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631798 2566 factory.go:223] Registration of the crio container factory successfully Jul 17 21:27:41.631921 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631853 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 21:27:41.631921 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631877 2566 factory.go:103] Registering Raw factory Jul 17 21:27:41.631921 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.631892 2566 manager.go:1196] Started watching for new ooms in manager Jul 17 21:27:41.632304 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.632287 2566 manager.go:349] Starting recovery of all containers Jul 17 21:27:41.640753 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.640554 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:41.644144 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.644119 2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-178.ec2.internal\" not found" node="ip-10-0-139-178.ec2.internal" Jul 17 21:27:41.645949 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.645931 2566 manager.go:354] Recovery completed Jul 17 21:27:41.651449 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.651434 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:41.653447 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.653428 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:41.653543 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.653472 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:41.653543 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.653488 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:41.654084 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.654072 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 21:27:41.654130 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.654084 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 21:27:41.654130 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.654092 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:41.654130 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.654101 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:27:41.654750 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.654735 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:41.654803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.654761 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:41.654803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.654772 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:41.655048 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.655037 2566 policy_none.go:49] "None policy: Start" Jul 17 21:27:41.655081 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.655054 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 21:27:41.655081 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.655063 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 21:27:41.655908 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.655898 2566 policy_none.go:47] "Start" Jul 17 21:27:41.705193 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.705175 2566 manager.go:337] "Starting Device Plugin manager" Jul 17 21:27:41.705303 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.705209 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 21:27:41.705303 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.705218 2566 server.go:87] "Starting device plugin registration server" Jul 17 21:27:41.705502 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.705479 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 21:27:41.705617 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.705497 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 21:27:41.705741 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.705713 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 21:27:41.705893 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.705881 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 21:27:41.705893 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.705894 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 21:27:41.706311 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.706234 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 21:27:41.706311 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.706293 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:41.805920 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.805841 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:41.809228 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.807128 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:41.809228 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.807166 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:41.809228 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.807177 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:41.809228 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.807202 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-178.ec2.internal" Jul 17 21:27:41.814595 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.814577 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-178.ec2.internal" Jul 17 21:27:41.814673 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.814621 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-178.ec2.internal\": node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:41.827926 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.827905 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:41.834658 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.834635 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 21:27:41.836089 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.836068 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 21:27:41.836185 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.836110 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 21:27:41.836185 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.836133 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 21:27:41.836265 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.836171 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 21:27:41.838227 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.838200 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:41.928522 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.928489 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:41.936911 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.936872 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal"] Jul 17 21:27:41.937055 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.936954 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:41.938568 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.938552 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:41.938696 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.938584 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:41.938696 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.938620 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:41.941024 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.941011 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:41.941155 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.941140 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:41.941213 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.941179 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:41.942181 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.942162 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:41.942258 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.942197 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:41.942258 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.942208 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:41.942258 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.942165 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:41.942343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.942270 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:41.942343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.942280 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:41.944164 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.944151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" Jul 17 21:27:41.944223 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.944176 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:27:41.944938 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.944912 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:27:41.945027 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.944948 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:27:41.945027 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:41.944963 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:27:41.976485 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.976455 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-178.ec2.internal\" not found" node="ip-10-0-139-178.ec2.internal" Jul 17 21:27:41.980943 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:41.980927 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-178.ec2.internal\" not found" node="ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.029208 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.029177 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:42.033530 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.033512 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43656a741cf2850076bf8121cab6d7b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal\" (UID: \"43656a741cf2850076bf8121cab6d7b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.033615 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.033537 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43656a741cf2850076bf8121cab6d7b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal\" (UID: \"43656a741cf2850076bf8121cab6d7b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.033615 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.033554 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d7ce4753063ea67abc0aa6d62b0262b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-178.ec2.internal\" (UID: \"6d7ce4753063ea67abc0aa6d62b0262b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.129355 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.129290 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:42.134631 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.134615 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43656a741cf2850076bf8121cab6d7b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal\" (UID: \"43656a741cf2850076bf8121cab6d7b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.134701 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.134643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43656a741cf2850076bf8121cab6d7b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal\" (UID: \"43656a741cf2850076bf8121cab6d7b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.134701 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.134660 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d7ce4753063ea67abc0aa6d62b0262b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-178.ec2.internal\" (UID: \"6d7ce4753063ea67abc0aa6d62b0262b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.134701 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.134684 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6d7ce4753063ea67abc0aa6d62b0262b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-178.ec2.internal\" (UID: \"6d7ce4753063ea67abc0aa6d62b0262b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.134701 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.134685 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43656a741cf2850076bf8121cab6d7b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal\" (UID: \"43656a741cf2850076bf8121cab6d7b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.134839 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.134699 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43656a741cf2850076bf8121cab6d7b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal\" (UID: \"43656a741cf2850076bf8121cab6d7b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.229832 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.229801 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:42.278273 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.278253 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.284825 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.284806 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" Jul 17 21:27:42.329921 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.329886 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:42.430406 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.430336 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:42.530810 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.530769 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:42.566103 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.566081 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 21:27:42.566698 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.566245 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 21:27:42.566698 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.566304 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 21:27:42.630991 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.630954 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:42.631148 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.630984 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:22:43 +0000 UTC" deadline="2028-01-19 10:00:48.416420368 +0000 UTC" Jul 17 21:27:42.631148 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:42.631042 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13212h33m5.785382216s" Jul 17 21:27:42.731096 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:42.731061 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:44.287166 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:44.287134 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:44.373292 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:44.373251 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7ce4753063ea67abc0aa6d62b0262b.slice/crio-a2cbd820a4d70a6a431c4c586234e2cbba08bae73052b117b5c22ceb2ed39b0c WatchSource:0}: Error finding container a2cbd820a4d70a6a431c4c586234e2cbba08bae73052b117b5c22ceb2ed39b0c: Status 404 returned error can't find the container with id a2cbd820a4d70a6a431c4c586234e2cbba08bae73052b117b5c22ceb2ed39b0c Jul 17 21:27:44.373736 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:44.373709 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43656a741cf2850076bf8121cab6d7b3.slice/crio-e4083541a9745831049e27b237ae1836aebada9411b56d0b1930ac0fdb228863 WatchSource:0}: Error finding container e4083541a9745831049e27b237ae1836aebada9411b56d0b1930ac0fdb228863: Status 404 returned error can't find the container with id e4083541a9745831049e27b237ae1836aebada9411b56d0b1930ac0fdb228863 Jul 17 21:27:44.378331 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.378314 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:27:44.387437 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:44.387418 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:44.465180 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.465100 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:44.488456 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:44.488429 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-178.ec2.internal\" not found" Jul 17 21:27:44.557308 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.557287 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:44.587373 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.587343 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" Jul 17 21:27:44.598579 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.598544 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:27:44.599758 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.599742 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" Jul 17 21:27:44.607656 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.607637 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:27:44.917590 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:44.917559 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:45.063625 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.063597 2566 apiserver.go:52] "Watching apiserver" Jul 17 21:27:45.072396 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.072364 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 21:27:45.072906 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.072872 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-g4x5v","openshift-network-diagnostics/network-check-target-467kv","kube-system/global-pull-secret-syncer-xbgxj","kube-system/konnectivity-agent-gjgzw","kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal","openshift-dns/node-resolver-5zrcf","openshift-image-registry/node-ca-52cmr","openshift-multus/multus-additional-cni-plugins-kfh2f","openshift-network-operator/iptables-alerter-ll5ws","openshift-ovn-kubernetes/ovnkube-node-phlpq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz","openshift-cluster-node-tuning-operator/tuned-hn98t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal","openshift-multus/multus-vphpc"] Jul 17 21:27:45.075548 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.075526 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.079135 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.078173 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.079135 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.078410 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 21:27:45.079135 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.078564 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 21:27:45.079135 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.078961 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-762kt\"" Jul 17 21:27:45.079135 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.078993 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 21:27:45.080443 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.080423 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.080568 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.080496 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:45.083380 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.082584 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 21:27:45.083380 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.082743 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 21:27:45.083380 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.082862 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 21:27:45.083380 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.083048 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 21:27:45.083380 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.083216 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c9rvn\"" Jul 17 21:27:45.083380 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.083349 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 21:27:45.083746 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.083536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.086070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.085638 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:45.086070 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.085706 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:45.086070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.085915 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 21:27:45.086070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.085966 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 21:27:45.086070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.086048 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pl4bj\"" Jul 17 21:27:45.088932 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.087026 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 21:27:45.091951 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.091283 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.094409 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.094255 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hmqpp\"" Jul 17 21:27:45.095454 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.094511 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 21:27:45.095454 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.094663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.095454 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.095111 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 21:27:45.096928 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.096902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.097012 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.096954 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7c6v\" (UniqueName: \"kubernetes.io/projected/45e3a6c9-bb86-424c-8909-feafef42373b-kube-api-access-k7c6v\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.097012 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.096994 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-os-release\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.097091 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.097091 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhscm\" (UniqueName: \"kubernetes.io/projected/29753f77-c24d-42c7-8518-24fd9698d3ac-kube-api-access-lhscm\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:45.097155 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45e3a6c9-bb86-424c-8909-feafef42373b-host\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.097155 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45e3a6c9-bb86-424c-8909-feafef42373b-serviceca\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.097155 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-system-cni-dir\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.097288 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097177 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f6911d6-a498-4c63-bb37-156b93e3a189-dbus\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.097288 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097206 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/780726fe-2e6b-4033-99a1-bf74fd25d7c7-agent-certs\") pod \"konnectivity-agent-gjgzw\" (UID: \"780726fe-2e6b-4033-99a1-bf74fd25d7c7\") " pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.097288 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097255 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/780726fe-2e6b-4033-99a1-bf74fd25d7c7-konnectivity-ca\") pod \"konnectivity-agent-gjgzw\" (UID: \"780726fe-2e6b-4033-99a1-bf74fd25d7c7\") " pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.097288 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097282 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c44a901-fbcb-40f0-9984-123b95d7dbb5-hosts-file\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.097439 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvq5\" (UniqueName: \"kubernetes.io/projected/1c44a901-fbcb-40f0-9984-123b95d7dbb5-kube-api-access-kzvq5\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.097439 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097348 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-cnibin\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.097439 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.097439 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f6911d6-a498-4c63-bb37-156b93e3a189-kubelet-config\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.097439 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:45.097681 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097466 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c44a901-fbcb-40f0-9984-123b95d7dbb5-tmp-dir\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.097681 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.097681 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.097681 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.097555 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678sx\" (UniqueName: \"kubernetes.io/projected/302e487b-c564-42d3-90d7-a846a77e2490-kube-api-access-678sx\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.099602 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.098311 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:27:45.099602 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.098588 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 21:27:45.099602 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.098801 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2vcpd\"" Jul 17 21:27:45.099602 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.098975 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 21:27:45.099602 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.099196 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.102073 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.100331 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:27:45.102073 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.101415 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:45.102073 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.101500 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:27:45.103140 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.103120 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.105008 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.104989 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.105253 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.105213 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 21:27:45.105507 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.105483 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hpncw\"" Jul 17 21:27:45.105933 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.105862 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 21:27:45.106103 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.106085 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bnf27\"" Jul 17 21:27:45.107168 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.106924 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 21:27:45.107261 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.107182 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.107574 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.107319 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 21:27:45.107574 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.107466 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 21:27:45.107574 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.107479 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 21:27:45.108392 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.108372 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 21:27:45.108489 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.108417 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 21:27:45.108489 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.108383 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 21:27:45.108594 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.108383 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 21:27:45.108715 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.108700 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wpfx\"" Jul 17 21:27:45.111324 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.109595 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:27:45.111324 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.109905 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f2br9\"" Jul 17 21:27:45.111324 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.110124 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 21:27:45.121266 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.121223 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jg5mj" Jul 17 21:27:45.131796 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.131770 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jg5mj" Jul 17 21:27:45.188251 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.188167 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 21:27:45.198458 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.198596 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7c6v\" (UniqueName: \"kubernetes.io/projected/45e3a6c9-bb86-424c-8909-feafef42373b-kube-api-access-k7c6v\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.198596 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.198596 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198522 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-daemon-config\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.198596 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198547 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysctl-conf\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.198596 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198595 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-var-lib-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198641 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-tuned\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198664 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fw6\" (UniqueName: \"kubernetes.io/projected/512d7aa6-e846-4f44-8bd3-cc2549d6d065-kube-api-access-v9fw6\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/780726fe-2e6b-4033-99a1-bf74fd25d7c7-konnectivity-ca\") pod \"konnectivity-agent-gjgzw\" (UID: \"780726fe-2e6b-4033-99a1-bf74fd25d7c7\") " pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-run-netns\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198745 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-cni-netd\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198792 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-cni-bin\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c44a901-fbcb-40f0-9984-123b95d7dbb5-tmp-dir\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.198849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198850 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-systemd-units\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198873 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198936 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-kubelet\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-log-socket\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.198983 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199010 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-device-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199035 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-iptables-alerter-script\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199055 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-k8s-cni-cncf-io\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199076 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-systemd\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199091 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-node-log\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199116 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdpkq\" (UniqueName: \"kubernetes.io/projected/2705ebe4-315f-420f-95a0-f6cc16012ec9-kube-api-access-bdpkq\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199141 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-cni-multus\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-kubelet\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199187 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysctl-d\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199208 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-run\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.199334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199245 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-slash\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f6911d6-a498-4c63-bb37-156b93e3a189-dbus\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/780726fe-2e6b-4033-99a1-bf74fd25d7c7-agent-certs\") pod \"konnectivity-agent-gjgzw\" (UID: \"780726fe-2e6b-4033-99a1-bf74fd25d7c7\") " pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199321 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf64r\" (UniqueName: \"kubernetes.io/projected/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-kube-api-access-bf64r\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-multus-certs\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbwg\" (UniqueName: \"kubernetes.io/projected/546a56e5-7cfe-403b-8ca8-625a943fbfb1-kube-api-access-nhbwg\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-cnibin\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199474 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199499 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-ovn\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199523 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovnkube-script-lib\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-kubernetes\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199566 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-systemd\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199579 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-sys\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199596 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-var-lib-kubelet\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.199887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199612 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-678sx\" (UniqueName: \"kubernetes.io/projected/302e487b-c564-42d3-90d7-a846a77e2490-kube-api-access-678sx\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovnkube-config\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199649 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-conf-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/512d7aa6-e846-4f44-8bd3-cc2549d6d065-tmp\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-os-release\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45e3a6c9-bb86-424c-8909-feafef42373b-serviceca\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-cni-bin\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/546a56e5-7cfe-403b-8ca8-625a943fbfb1-cni-binary-copy\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199750 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-etc-kubernetes\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199772 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-host\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199802 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-system-cni-dir\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c44a901-fbcb-40f0-9984-123b95d7dbb5-hosts-file\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvq5\" (UniqueName: \"kubernetes.io/projected/1c44a901-fbcb-40f0-9984-123b95d7dbb5-kube-api-access-kzvq5\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199847 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-socket-dir-parent\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd82t\" (UniqueName: \"kubernetes.io/projected/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-kube-api-access-hd82t\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.200527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199925 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-system-cni-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199969 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-etc-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.199992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhscm\" (UniqueName: \"kubernetes.io/projected/29753f77-c24d-42c7-8518-24fd9698d3ac-kube-api-access-lhscm\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45e3a6c9-bb86-424c-8909-feafef42373b-host\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200025 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovn-node-metrics-cert\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200054 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-host-slash\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysconfig\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-env-overrides\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200134 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-os-release\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200148 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-netns\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200162 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-lib-modules\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f6911d6-a498-4c63-bb37-156b93e3a189-kubelet-config\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200195 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-cni-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200222 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-cnibin\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.201095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200263 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-hostroot\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.201640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-modprobe-d\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.201640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200932 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.201640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.200964 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-cni-binary-copy\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.201640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.201508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-os-release\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.201640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.201568 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/780726fe-2e6b-4033-99a1-bf74fd25d7c7-konnectivity-ca\") pod \"konnectivity-agent-gjgzw\" (UID: \"780726fe-2e6b-4033-99a1-bf74fd25d7c7\") " pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.201895 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.201879 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45e3a6c9-bb86-424c-8909-feafef42373b-serviceca\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.201974 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.201936 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c44a901-fbcb-40f0-9984-123b95d7dbb5-tmp-dir\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.201974 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.201956 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-system-cni-dir\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.202128 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.202093 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c44a901-fbcb-40f0-9984-123b95d7dbb5-hosts-file\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.202575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.202407 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f6911d6-a498-4c63-bb37-156b93e3a189-kubelet-config\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.202575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.202493 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.202575 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.202532 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:45.208867 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.208313 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f6911d6-a498-4c63-bb37-156b93e3a189-dbus\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.208867 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.208384 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45e3a6c9-bb86-424c-8909-feafef42373b-host\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.208867 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.202544 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/302e487b-c564-42d3-90d7-a846a77e2490-cnibin\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.208867 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.208451 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs podName:29753f77-c24d-42c7-8518-24fd9698d3ac nodeName:}" failed. No retries permitted until 2026-07-17 21:27:45.708405079 +0000 UTC m=+2.978402783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs") pod "network-metrics-daemon-g4x5v" (UID: "29753f77-c24d-42c7-8518-24fd9698d3ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:45.208867 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.208820 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:45.208867 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.208823 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/302e487b-c564-42d3-90d7-a846a77e2490-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.209275 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.208892 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret podName:4f6911d6-a498-4c63-bb37-156b93e3a189 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:45.708875263 +0000 UTC m=+2.978872961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret") pod "global-pull-secret-syncer-xbgxj" (UID: "4f6911d6-a498-4c63-bb37-156b93e3a189") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:45.209275 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.209058 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 21:27:45.214213 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.214183 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/780726fe-2e6b-4033-99a1-bf74fd25d7c7-agent-certs\") pod \"konnectivity-agent-gjgzw\" (UID: \"780726fe-2e6b-4033-99a1-bf74fd25d7c7\") " pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.214345 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.214189 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-678sx\" (UniqueName: \"kubernetes.io/projected/302e487b-c564-42d3-90d7-a846a77e2490-kube-api-access-678sx\") pod \"multus-additional-cni-plugins-kfh2f\" (UID: \"302e487b-c564-42d3-90d7-a846a77e2490\") " pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.214587 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.214565 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7c6v\" (UniqueName: \"kubernetes.io/projected/45e3a6c9-bb86-424c-8909-feafef42373b-kube-api-access-k7c6v\") pod \"node-ca-52cmr\" (UID: \"45e3a6c9-bb86-424c-8909-feafef42373b\") " pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.214829 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.214809 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvq5\" (UniqueName: \"kubernetes.io/projected/1c44a901-fbcb-40f0-9984-123b95d7dbb5-kube-api-access-kzvq5\") pod \"node-resolver-5zrcf\" (UID: \"1c44a901-fbcb-40f0-9984-123b95d7dbb5\") " pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.217669 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.217628 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhscm\" (UniqueName: \"kubernetes.io/projected/29753f77-c24d-42c7-8518-24fd9698d3ac-kube-api-access-lhscm\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:45.297075 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.297016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" event={"ID":"6d7ce4753063ea67abc0aa6d62b0262b","Type":"ContainerStarted","Data":"a2cbd820a4d70a6a431c4c586234e2cbba08bae73052b117b5c22ceb2ed39b0c"} Jul 17 21:27:45.299118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.299086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" event={"ID":"43656a741cf2850076bf8121cab6d7b3","Type":"ContainerStarted","Data":"e4083541a9745831049e27b237ae1836aebada9411b56d0b1930ac0fdb228863"} Jul 17 21:27:45.301523 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301494 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-cni-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.301634 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-cnibin\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.301634 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-hostroot\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.301634 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-modprobe-d\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.301634 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.301634 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-daemon-config\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.301819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301653 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysctl-conf\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.301819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-var-lib-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.301819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-tuned\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.301819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301722 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fw6\" (UniqueName: \"kubernetes.io/projected/512d7aa6-e846-4f44-8bd3-cc2549d6d065-kube-api-access-v9fw6\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.301819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301745 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-run-netns\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.301819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-cni-netd\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.301819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301801 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301824 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-cni-bin\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-systemd-units\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301895 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-kubelet\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301917 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-log-socket\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-device-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.301992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-iptables-alerter-script\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.302026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-k8s-cni-cncf-io\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302039 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-systemd\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-node-log\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302092 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdpkq\" (UniqueName: \"kubernetes.io/projected/2705ebe4-315f-420f-95a0-f6cc16012ec9-kube-api-access-bdpkq\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302115 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-cni-multus\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302138 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-kubelet\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302164 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysctl-d\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-run\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-slash\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302268 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf64r\" (UniqueName: \"kubernetes.io/projected/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-kube-api-access-bf64r\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302318 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-hostroot\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302397 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-cni-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysctl-conf\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302424 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-systemd-units\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302448 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-cnibin\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302461 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-var-lib-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302498 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-systemd\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302499 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302525 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302540 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-kubelet\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302589 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-node-log\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-modprobe-d\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302632 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-run\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302641 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-device-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.302689 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-kubelet\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302704 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-log-socket\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302701 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302743 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-cni-netd\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302791 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-slash\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302914 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-run-netns\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.302971 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-iptables-alerter-script\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303009 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-k8s-cni-cncf-io\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303034 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysctl-d\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303073 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-cni-bin\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303174 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-daemon-config\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303179 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303244 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-multus-certs\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303266 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303273 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbwg\" (UniqueName: \"kubernetes.io/projected/546a56e5-7cfe-403b-8ca8-625a943fbfb1-kube-api-access-nhbwg\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303276 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-var-lib-cni-multus\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.303299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303304 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-multus-certs\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303304 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-ovn\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303341 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovnkube-script-lib\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303339 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-ovn\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-kubernetes\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303382 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-systemd\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303411 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-systemd\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303447 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-kubernetes\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-sys\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-var-lib-kubelet\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovnkube-config\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303558 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-conf-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-sys\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303580 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/512d7aa6-e846-4f44-8bd3-cc2549d6d065-tmp\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303595 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-var-lib-kubelet\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-cni-bin\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-host-cni-bin\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/546a56e5-7cfe-403b-8ca8-625a943fbfb1-cni-binary-copy\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303683 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-conf-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303883 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-etc-kubernetes\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303898 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovnkube-script-lib\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303921 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-host\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-etc-kubernetes\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-socket-dir-parent\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303995 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.303997 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-host\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304041 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-multus-socket-dir-parent\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304052 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-run-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd82t\" (UniqueName: \"kubernetes.io/projected/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-kube-api-access-hd82t\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304095 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-system-cni-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-etc-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304143 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovn-node-metrics-cert\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304168 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovnkube-config\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304251 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-system-cni-dir\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304265 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/546a56e5-7cfe-403b-8ca8-625a943fbfb1-cni-binary-copy\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304292 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-etc-openvswitch\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.304897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304374 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-host-slash\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304387 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304400 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysconfig\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304436 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-host-slash\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304448 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-sysconfig\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-env-overrides\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304508 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-os-release\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-netns\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304571 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-host-run-netns\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-lib-modules\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/546a56e5-7cfe-403b-8ca8-625a943fbfb1-os-release\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304657 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2705ebe4-315f-420f-95a0-f6cc16012ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304721 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/512d7aa6-e846-4f44-8bd3-cc2549d6d065-lib-modules\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.304856 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-env-overrides\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.305971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.305588 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/512d7aa6-e846-4f44-8bd3-cc2549d6d065-etc-tuned\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.306719 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.306688 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/512d7aa6-e846-4f44-8bd3-cc2549d6d065-tmp\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.307150 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.307128 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-ovn-node-metrics-cert\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.311027 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.310583 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:45.311027 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.310614 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:45.311027 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.310630 2566 projected.go:196] Error preparing data for projected volume kube-api-access-kswmr for pod openshift-network-diagnostics/network-check-target-467kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:45.311027 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.310705 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr podName:b3219454-f5d7-41d3-8ad9-662ae16c50b2 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:45.810687469 +0000 UTC m=+3.080685152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kswmr" (UniqueName: "kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr") pod "network-check-target-467kv" (UID: "b3219454-f5d7-41d3-8ad9-662ae16c50b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:45.313460 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.313109 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbwg\" (UniqueName: \"kubernetes.io/projected/546a56e5-7cfe-403b-8ca8-625a943fbfb1-kube-api-access-nhbwg\") pod \"multus-vphpc\" (UID: \"546a56e5-7cfe-403b-8ca8-625a943fbfb1\") " pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.313758 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.313710 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdpkq\" (UniqueName: \"kubernetes.io/projected/2705ebe4-315f-420f-95a0-f6cc16012ec9-kube-api-access-bdpkq\") pod \"aws-ebs-csi-driver-node-2bgxz\" (UID: \"2705ebe4-315f-420f-95a0-f6cc16012ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.314632 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.314612 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fw6\" (UniqueName: \"kubernetes.io/projected/512d7aa6-e846-4f44-8bd3-cc2549d6d065-kube-api-access-v9fw6\") pod \"tuned-hn98t\" (UID: \"512d7aa6-e846-4f44-8bd3-cc2549d6d065\") " pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.315463 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.315417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf64r\" (UniqueName: \"kubernetes.io/projected/e3527bbd-d893-4732-a3a5-8f87cfe5f8b4-kube-api-access-bf64r\") pod \"ovnkube-node-phlpq\" (UID: \"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.316671 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.316650 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd82t\" (UniqueName: \"kubernetes.io/projected/4f4e675b-75c7-4d39-8853-0d5dd4b54fed-kube-api-access-hd82t\") pod \"iptables-alerter-ll5ws\" (UID: \"4f4e675b-75c7-4d39-8853-0d5dd4b54fed\") " pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.393830 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.393792 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-52cmr" Jul 17 21:27:45.406137 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.406096 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" Jul 17 21:27:45.416897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.416874 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:27:45.422602 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.422583 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zrcf" Jul 17 21:27:45.431077 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.431058 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ll5ws" Jul 17 21:27:45.440007 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.439823 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vphpc" Jul 17 21:27:45.448592 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.448571 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:27:45.455411 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.455391 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" Jul 17 21:27:45.461181 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.461158 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hn98t" Jul 17 21:27:45.563891 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.563859 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:27:45.808621 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.808534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:45.808621 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.808576 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:45.808839 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.808722 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:45.808839 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.808730 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:45.808839 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.808783 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret podName:4f6911d6-a498-4c63-bb37-156b93e3a189 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:46.808767841 +0000 UTC m=+4.078765530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret") pod "global-pull-secret-syncer-xbgxj" (UID: "4f6911d6-a498-4c63-bb37-156b93e3a189") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:45.808839 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.808804 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs podName:29753f77-c24d-42c7-8518-24fd9698d3ac nodeName:}" failed. No retries permitted until 2026-07-17 21:27:46.808793623 +0000 UTC m=+4.078791322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs") pod "network-metrics-daemon-g4x5v" (UID: "29753f77-c24d-42c7-8518-24fd9698d3ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:45.908975 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:45.908927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:45.909148 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.909093 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:45.909148 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.909115 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:45.909148 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.909127 2566 projected.go:196] Error preparing data for projected volume kube-api-access-kswmr for pod openshift-network-diagnostics/network-check-target-467kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:45.909302 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:45.909190 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr podName:b3219454-f5d7-41d3-8ad9-662ae16c50b2 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:46.909173957 +0000 UTC m=+4.179171640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kswmr" (UniqueName: "kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr") pod "network-check-target-467kv" (UID: "b3219454-f5d7-41d3-8ad9-662ae16c50b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:46.134992 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:46.134857 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:22:45 +0000 UTC" deadline="2027-12-23 14:58:08.284779567 +0000 UTC" Jul 17 21:27:46.134992 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:46.134948 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12569h30m22.149855962s" Jul 17 21:27:46.292622 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:46.292591 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:46.292769 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.292703 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:46.292769 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:46.292733 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:46.292855 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.292803 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:46.368286 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:46.367597 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4e675b_75c7_4d39_8853_0d5dd4b54fed.slice/crio-77e4a360202a8e59972d2da3effa71e317a14bdcfd2c43aba323a771158e9d3d WatchSource:0}: Error finding container 77e4a360202a8e59972d2da3effa71e317a14bdcfd2c43aba323a771158e9d3d: Status 404 returned error can't find the container with id 77e4a360202a8e59972d2da3effa71e317a14bdcfd2c43aba323a771158e9d3d Jul 17 21:27:46.368286 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:46.368186 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512d7aa6_e846_4f44_8bd3_cc2549d6d065.slice/crio-0090e70b3ee6c4d5636191efa8d45d6f76467f4e45dcab2e2d1b227e7022f118 WatchSource:0}: Error finding container 0090e70b3ee6c4d5636191efa8d45d6f76467f4e45dcab2e2d1b227e7022f118: Status 404 returned error can't find the container with id 0090e70b3ee6c4d5636191efa8d45d6f76467f4e45dcab2e2d1b227e7022f118 Jul 17 21:27:46.373375 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:46.373352 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2705ebe4_315f_420f_95a0_f6cc16012ec9.slice/crio-cc63eac059e923ae02205c7559424e6ef299df3f6b976be25982d9af1a6b62df WatchSource:0}: Error finding container cc63eac059e923ae02205c7559424e6ef299df3f6b976be25982d9af1a6b62df: Status 404 returned error can't find the container with id cc63eac059e923ae02205c7559424e6ef299df3f6b976be25982d9af1a6b62df Jul 17 21:27:46.374516 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:46.374497 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302e487b_c564_42d3_90d7_a846a77e2490.slice/crio-2464cd455905c0ef2d111c054f396a4646e255bed2b5146f53843dc8f2e60609 WatchSource:0}: Error finding container 2464cd455905c0ef2d111c054f396a4646e255bed2b5146f53843dc8f2e60609: Status 404 returned error can't find the container with id 2464cd455905c0ef2d111c054f396a4646e255bed2b5146f53843dc8f2e60609 Jul 17 21:27:46.375395 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:27:46.375375 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546a56e5_7cfe_403b_8ca8_625a943fbfb1.slice/crio-48c79a9eac261c3b9e43b9238ca84669999b4329cbd324459baa9296cb2dbe0a WatchSource:0}: Error finding container 48c79a9eac261c3b9e43b9238ca84669999b4329cbd324459baa9296cb2dbe0a: Status 404 returned error can't find the container with id 48c79a9eac261c3b9e43b9238ca84669999b4329cbd324459baa9296cb2dbe0a Jul 17 21:27:46.818869 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:46.818122 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:46.818869 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:46.818169 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:46.818869 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.818327 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:46.818869 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.818389 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret podName:4f6911d6-a498-4c63-bb37-156b93e3a189 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:48.818371648 +0000 UTC m=+6.088369337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret") pod "global-pull-secret-syncer-xbgxj" (UID: "4f6911d6-a498-4c63-bb37-156b93e3a189") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:46.818869 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.818791 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:46.818869 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.818836 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs podName:29753f77-c24d-42c7-8518-24fd9698d3ac nodeName:}" failed. No retries permitted until 2026-07-17 21:27:48.818820788 +0000 UTC m=+6.088818473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs") pod "network-metrics-daemon-g4x5v" (UID: "29753f77-c24d-42c7-8518-24fd9698d3ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:46.919125 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:46.919084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:46.919334 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.919316 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:46.919422 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.919346 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:46.919422 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.919359 2566 projected.go:196] Error preparing data for projected volume kube-api-access-kswmr for pod openshift-network-diagnostics/network-check-target-467kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:46.919526 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:46.919420 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr podName:b3219454-f5d7-41d3-8ad9-662ae16c50b2 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:48.91940142 +0000 UTC m=+6.189399120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kswmr" (UniqueName: "kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr") pod "network-check-target-467kv" (UID: "b3219454-f5d7-41d3-8ad9-662ae16c50b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:47.135611 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.135498 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:22:45 +0000 UTC" deadline="2028-03-17 02:06:56.228139663 +0000 UTC" Jul 17 21:27:47.135611 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.135560 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14596h39m9.092584281s" Jul 17 21:27:47.295108 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.294623 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:47.295108 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:47.294745 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:27:47.317090 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.317045 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zrcf" event={"ID":"1c44a901-fbcb-40f0-9984-123b95d7dbb5","Type":"ContainerStarted","Data":"f66279422af9b564836788667288326963d26fd7c0c1b4c691558472ccb96a18"} Jul 17 21:27:47.329023 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.328913 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"654d24f3443a3388a6b87946db4ac9dfaf07031dd55856766f1783c9e2d44aeb"} Jul 17 21:27:47.337096 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.337005 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vphpc" event={"ID":"546a56e5-7cfe-403b-8ca8-625a943fbfb1","Type":"ContainerStarted","Data":"48c79a9eac261c3b9e43b9238ca84669999b4329cbd324459baa9296cb2dbe0a"} Jul 17 21:27:47.339550 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.339519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerStarted","Data":"2464cd455905c0ef2d111c054f396a4646e255bed2b5146f53843dc8f2e60609"} Jul 17 21:27:47.344680 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.344648 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hn98t" event={"ID":"512d7aa6-e846-4f44-8bd3-cc2549d6d065","Type":"ContainerStarted","Data":"0090e70b3ee6c4d5636191efa8d45d6f76467f4e45dcab2e2d1b227e7022f118"} Jul 17 21:27:47.362142 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.361404 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" event={"ID":"6d7ce4753063ea67abc0aa6d62b0262b","Type":"ContainerStarted","Data":"3a7dd5d2ddd2eeb1faa890e35312bee12019fb4c9e26b085a399b89cdfbe5863"} Jul 17 21:27:47.367098 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.366471 2566 generic.go:356] "Generic (PLEG): container finished" podID="43656a741cf2850076bf8121cab6d7b3" containerID="28d847ac71b4efa8f49324cbe5f4cbd273d55fadfafb9e82285c8941b8ab3299" exitCode=0 Jul 17 21:27:47.367098 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.366559 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" event={"ID":"43656a741cf2850076bf8121cab6d7b3","Type":"ContainerDied","Data":"28d847ac71b4efa8f49324cbe5f4cbd273d55fadfafb9e82285c8941b8ab3299"} Jul 17 21:27:47.377628 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.376627 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-178.ec2.internal" podStartSLOduration=3.37661008 podStartE2EDuration="3.37661008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:47.376471445 +0000 UTC m=+4.646469150" watchObservedRunningTime="2026-07-17 21:27:47.37661008 +0000 UTC m=+4.646607791" Jul 17 21:27:47.385568 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.385541 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjgzw" event={"ID":"780726fe-2e6b-4033-99a1-bf74fd25d7c7","Type":"ContainerStarted","Data":"80bb6f1006a03eb549c2d3e7ffa1966614dc190211aae493fec64a7f54e8efe3"} Jul 17 21:27:47.387422 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.387397 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" event={"ID":"2705ebe4-315f-420f-95a0-f6cc16012ec9","Type":"ContainerStarted","Data":"cc63eac059e923ae02205c7559424e6ef299df3f6b976be25982d9af1a6b62df"} Jul 17 21:27:47.400635 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.400602 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ll5ws" event={"ID":"4f4e675b-75c7-4d39-8853-0d5dd4b54fed","Type":"ContainerStarted","Data":"77e4a360202a8e59972d2da3effa71e317a14bdcfd2c43aba323a771158e9d3d"} Jul 17 21:27:47.404300 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:47.404272 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-52cmr" event={"ID":"45e3a6c9-bb86-424c-8909-feafef42373b","Type":"ContainerStarted","Data":"82fca7832b21295b6549f014eafc27a41dbe6e59f3e1a22fc27bf9c3a7fe5133"} Jul 17 21:27:48.293026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:48.292986 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:48.293569 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.293120 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:48.293569 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:48.292992 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:48.293694 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.293592 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:48.419150 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:48.418757 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" event={"ID":"43656a741cf2850076bf8121cab6d7b3","Type":"ContainerStarted","Data":"785997e9b5ba60322c133bec98f59093cf5b5de3fe39506f518ab3a8b80d257a"} Jul 17 21:27:48.448290 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:48.447276 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-178.ec2.internal" podStartSLOduration=4.447254713 podStartE2EDuration="4.447254713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:48.446727186 +0000 UTC m=+5.716724870" watchObservedRunningTime="2026-07-17 21:27:48.447254713 +0000 UTC m=+5.717252409" Jul 17 21:27:48.833993 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:48.833917 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:48.833993 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:48.833969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:48.834260 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.834130 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:48.834260 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.834195 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret podName:4f6911d6-a498-4c63-bb37-156b93e3a189 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:52.83417815 +0000 UTC m=+10.104175862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret") pod "global-pull-secret-syncer-xbgxj" (UID: "4f6911d6-a498-4c63-bb37-156b93e3a189") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:48.834667 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.834644 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:48.834766 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.834702 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs podName:29753f77-c24d-42c7-8518-24fd9698d3ac nodeName:}" failed. No retries permitted until 2026-07-17 21:27:52.834686429 +0000 UTC m=+10.104684117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs") pod "network-metrics-daemon-g4x5v" (UID: "29753f77-c24d-42c7-8518-24fd9698d3ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:48.935973 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:48.935328 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:48.935973 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.935538 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:48.935973 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.935560 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:48.935973 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.935571 2566 projected.go:196] Error preparing data for projected volume kube-api-access-kswmr for pod openshift-network-diagnostics/network-check-target-467kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:48.935973 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:48.935614 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr podName:b3219454-f5d7-41d3-8ad9-662ae16c50b2 nodeName:}" failed. No retries permitted until 2026-07-17 21:27:52.935600348 +0000 UTC m=+10.205598030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kswmr" (UniqueName: "kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr") pod "network-check-target-467kv" (UID: "b3219454-f5d7-41d3-8ad9-662ae16c50b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:49.293363 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:49.293332 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:49.293854 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:49.293479 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:27:50.292704 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:50.292621 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:50.292890 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:50.292730 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:50.293038 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:50.293021 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:50.293114 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:50.293094 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:51.292290 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:51.292077 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:51.292290 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:51.292209 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:27:52.292383 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:52.292345 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:52.292812 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.292474 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:52.292901 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:52.292846 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:52.292977 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.292944 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:52.867846 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:52.867799 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:52.868029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:52.867861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:52.868029 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.867990 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:52.868116 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.868035 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:52.868116 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.868064 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs podName:29753f77-c24d-42c7-8518-24fd9698d3ac nodeName:}" failed. No retries permitted until 2026-07-17 21:28:00.868042481 +0000 UTC m=+18.138040163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs") pod "network-metrics-daemon-g4x5v" (UID: "29753f77-c24d-42c7-8518-24fd9698d3ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:27:52.868116 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.868089 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret podName:4f6911d6-a498-4c63-bb37-156b93e3a189 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:00.868072364 +0000 UTC m=+18.138070046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret") pod "global-pull-secret-syncer-xbgxj" (UID: "4f6911d6-a498-4c63-bb37-156b93e3a189") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:27:52.969068 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:52.968967 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:52.969326 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.969134 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:27:52.969326 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.969158 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:27:52.969326 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.969172 2566 projected.go:196] Error preparing data for projected volume kube-api-access-kswmr for pod openshift-network-diagnostics/network-check-target-467kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:52.969326 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:52.969254 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr podName:b3219454-f5d7-41d3-8ad9-662ae16c50b2 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:00.969218123 +0000 UTC m=+18.239215804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kswmr" (UniqueName: "kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr") pod "network-check-target-467kv" (UID: "b3219454-f5d7-41d3-8ad9-662ae16c50b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:27:53.293498 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:53.293469 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:53.293967 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:53.293565 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:27:54.292731 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:54.292698 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:54.292900 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:54.292750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:54.292900 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:54.292840 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:54.292989 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:54.292960 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:55.292266 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:55.292213 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:55.292842 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:55.292355 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:27:56.292646 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:56.292611 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:56.293097 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:56.292660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:56.293097 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:56.292745 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:56.293097 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:56.292846 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:57.292436 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:57.292399 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:57.292621 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:57.292534 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:27:58.292157 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:58.292121 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:27:58.292546 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:58.292245 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:27:58.292546 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:58.292260 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:27:58.292546 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:58.292321 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:27:59.292360 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:27:59.292314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:27:59.292832 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:27:59.292453 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:00.292652 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:00.292616 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:00.293104 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:00.292762 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:00.293104 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:00.292780 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:00.293104 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:00.292855 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:00.933424 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:00.933392 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:00.933424 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:00.933428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:00.933591 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:00.933538 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:28:00.933591 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:00.933549 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:28:00.933703 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:00.933593 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret podName:4f6911d6-a498-4c63-bb37-156b93e3a189 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:16.933579931 +0000 UTC m=+34.203577613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret") pod "global-pull-secret-syncer-xbgxj" (UID: "4f6911d6-a498-4c63-bb37-156b93e3a189") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:28:00.933703 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:00.933607 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs podName:29753f77-c24d-42c7-8518-24fd9698d3ac nodeName:}" failed. No retries permitted until 2026-07-17 21:28:16.933600117 +0000 UTC m=+34.203597799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs") pod "network-metrics-daemon-g4x5v" (UID: "29753f77-c24d-42c7-8518-24fd9698d3ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:28:01.034397 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:01.034369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:01.035018 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:01.035003 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:28:01.035063 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:01.035024 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:28:01.035063 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:01.035033 2566 projected.go:196] Error preparing data for projected volume kube-api-access-kswmr for pod openshift-network-diagnostics/network-check-target-467kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:28:01.035139 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:01.035081 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr podName:b3219454-f5d7-41d3-8ad9-662ae16c50b2 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:17.035065988 +0000 UTC m=+34.305063669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kswmr" (UniqueName: "kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr") pod "network-check-target-467kv" (UID: "b3219454-f5d7-41d3-8ad9-662ae16c50b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:28:01.292745 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:01.292704 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:01.293191 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:01.292839 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:02.292306 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:02.292269 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:02.292498 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:02.292269 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:02.292498 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:02.292400 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:02.292603 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:02.292518 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:03.293995 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:03.293956 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:03.294419 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:03.294073 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:04.292724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:04.292689 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:04.292724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:04.292709 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:04.292959 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:04.292827 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:04.293023 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:04.292982 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:05.292930 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:05.292900 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:05.293559 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:05.293002 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:06.292904 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.292585 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:06.293146 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.292638 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:06.293146 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:06.292992 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:06.293146 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:06.293052 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:06.457926 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.457890 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjgzw" event={"ID":"780726fe-2e6b-4033-99a1-bf74fd25d7c7","Type":"ContainerStarted","Data":"20e4cfb7ae47c120142c66c45db85dffb4c00b9177fca174d8d2f0185533fcf4"} Jul 17 21:28:06.459283 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.459256 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" event={"ID":"2705ebe4-315f-420f-95a0-f6cc16012ec9","Type":"ContainerStarted","Data":"a2cacf47c20e287201665f663f53d41258088371ed4d3b0043f10cf80c9ace8a"} Jul 17 21:28:06.460441 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.460416 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-52cmr" event={"ID":"45e3a6c9-bb86-424c-8909-feafef42373b","Type":"ContainerStarted","Data":"56d957b7f5610d56fe0d38bcd41d2456ddb46a89165c91849898d8b2aace6497"} Jul 17 21:28:06.461617 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.461595 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zrcf" event={"ID":"1c44a901-fbcb-40f0-9984-123b95d7dbb5","Type":"ContainerStarted","Data":"63bee77e80ac0854054c63a8b6e73d73803dca9ff489b5d2987f8afe1df690b7"} Jul 17 21:28:06.465690 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.465668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"db5fa7ff83c03766f2d6227b0f973a68394fee704fae773f2ec7dfcb45445a60"} Jul 17 21:28:06.465799 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.465742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"01bc88f4f935d6688494306bf5926d5b4886d75f0e21ddf02075418c8c0e01e1"} Jul 17 21:28:06.465799 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.465795 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"63bb92d2dc60d3069b70eb759283a657a4cfa61d0bf8cf436ee54f7895131b80"} Jul 17 21:28:06.465913 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.465841 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"5622aa367d39f154758a910702eae9b4d58db284bc3bca2a3cb7e4bc2419966d"} Jul 17 21:28:06.465913 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.465892 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"a0fd056d5c2d53c94ce8da4b7150ee960d21fe9f61de934b2c59cfebe8cd93b2"} Jul 17 21:28:06.466008 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.465941 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"06e8eb787e973c15b96e74bf8603a7fa8831c2cb69121100fc3c73955c9486e7"} Jul 17 21:28:06.466924 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.466905 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vphpc" event={"ID":"546a56e5-7cfe-403b-8ca8-625a943fbfb1","Type":"ContainerStarted","Data":"6a9dee469c89224e1ba16f1befb96977af9482a91fec7f533898d71598cf78b0"} Jul 17 21:28:06.468296 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.468276 2566 generic.go:356] "Generic (PLEG): container finished" podID="302e487b-c564-42d3-90d7-a846a77e2490" containerID="8fba4cc35a93fec83ead98dc99d3edb9276050085ff5d28d96b945a7766efcaa" exitCode=0 Jul 17 21:28:06.468376 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.468338 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerDied","Data":"8fba4cc35a93fec83ead98dc99d3edb9276050085ff5d28d96b945a7766efcaa"} Jul 17 21:28:06.469665 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.469642 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hn98t" event={"ID":"512d7aa6-e846-4f44-8bd3-cc2549d6d065","Type":"ContainerStarted","Data":"fa12bdcf0f604e092d78bc95540e15889ecc0f0af4e721c361945da1cd07e96c"} Jul 17 21:28:06.479126 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.479087 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gjgzw" podStartSLOduration=9.002488892 podStartE2EDuration="23.479075764s" totalImagesPullingTime="14.476586872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:06.478902151 +0000 UTC m=+23.748899855" watchObservedRunningTime="2026-07-17 21:28:06.479075764 +0000 UTC m=+23.749073525" Jul 17 21:28:06.497375 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.497318 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5zrcf" podStartSLOduration=4.577797451 podStartE2EDuration="23.497304774s" totalImagesPullingTime="18.919507323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:06.49706274 +0000 UTC m=+23.767060443" watchObservedRunningTime="2026-07-17 21:28:06.497304774 +0000 UTC m=+23.767302478" Jul 17 21:28:06.517376 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.517336 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hn98t" podStartSLOduration=4.5603874189999996 podStartE2EDuration="23.517322483s" totalImagesPullingTime="18.956935064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:06.517049433 +0000 UTC m=+23.787047135" watchObservedRunningTime="2026-07-17 21:28:06.517322483 +0000 UTC m=+23.787320189" Jul 17 21:28:06.534302 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.534253 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-52cmr" podStartSLOduration=4.622558931 podStartE2EDuration="23.534215639s" totalImagesPullingTime="18.911656708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:06.534199507 +0000 UTC m=+23.804197210" watchObservedRunningTime="2026-07-17 21:28:06.534215639 +0000 UTC m=+23.804213344" Jul 17 21:28:06.581783 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.581742 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vphpc" podStartSLOduration=4.607349474 podStartE2EDuration="23.581726628s" totalImagesPullingTime="18.974377154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:06.581191902 +0000 UTC m=+23.851189606" watchObservedRunningTime="2026-07-17 21:28:06.581726628 +0000 UTC m=+23.851724332" Jul 17 21:28:06.808678 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:06.808654 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 21:28:07.179490 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:07.179371 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:28:06.808672221Z","UUID":"7527561e-73fb-46b8-9aa9-6f4bcdc8c550","Handler":null,"Name":"","Endpoint":""} Jul 17 21:28:07.180934 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:07.180915 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 21:28:07.181077 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:07.180954 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 21:28:07.292971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:07.292927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:07.293149 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:07.293067 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:07.475375 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:07.475278 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" event={"ID":"2705ebe4-315f-420f-95a0-f6cc16012ec9","Type":"ContainerStarted","Data":"a1bcd5620fdaec1c9b184847d174920ca8eb9771f68c461408bb334cc551ee38"} Jul 17 21:28:07.478149 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:07.478120 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ll5ws" event={"ID":"4f4e675b-75c7-4d39-8853-0d5dd4b54fed","Type":"ContainerStarted","Data":"8d242e0e345e9ccb561507b98f70133bd64b222159582dee3a52a4d380de400a"} Jul 17 21:28:07.495249 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:07.495169 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ll5ws" podStartSLOduration=5.562015868 podStartE2EDuration="24.495149678s" totalImagesPullingTime="18.93313381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:07.494550973 +0000 UTC m=+24.764548876" watchObservedRunningTime="2026-07-17 21:28:07.495149678 +0000 UTC m=+24.765147383" Jul 17 21:28:08.292169 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:08.291920 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:08.292343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:08.291925 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:08.292343 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:08.292245 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:08.292343 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:08.292331 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:08.484346 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:08.484310 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"e62c33ee3e7f0d0cb3e0adccd6157d67fdf5da3bc6340ab4f2a75d124f4ee8fa"} Jul 17 21:28:08.486541 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:08.486516 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" event={"ID":"2705ebe4-315f-420f-95a0-f6cc16012ec9","Type":"ContainerStarted","Data":"e5db35f1da949845c5a2c28dd73cf9bede9f0b1631d4aeaf3c855995152ecadc"} Jul 17 21:28:08.504619 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:08.504566 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2bgxz" podStartSLOduration=4.288418077 podStartE2EDuration="25.504544662s" totalImagesPullingTime="21.216126585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:08.504128908 +0000 UTC m=+25.774126612" watchObservedRunningTime="2026-07-17 21:28:08.504544662 +0000 UTC m=+25.774542369" Jul 17 21:28:09.292012 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:09.291981 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:09.292212 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:09.292090 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:10.292059 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:10.292010 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:10.292564 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:10.292016 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:10.292564 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:10.292153 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:10.292564 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:10.292250 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:10.753941 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:10.753914 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:28:10.754605 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:10.754585 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:28:11.293280 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.293026 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:11.293987 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:11.293316 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:11.496697 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.496656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" event={"ID":"e3527bbd-d893-4732-a3a5-8f87cfe5f8b4","Type":"ContainerStarted","Data":"879cf58f44e48b011e6d0fafbdb4b20254760306ce89890a6f251543d73e5dd8"} Jul 17 21:28:11.496910 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.496895 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:28:11.498664 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.498641 2566 generic.go:356] "Generic (PLEG): container finished" podID="302e487b-c564-42d3-90d7-a846a77e2490" containerID="428b95a70caafd23a3c44ce211b5dd720e2390626fc96919d5e899b259d80d4e" exitCode=0 Jul 17 21:28:11.498764 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.498694 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerDied","Data":"428b95a70caafd23a3c44ce211b5dd720e2390626fc96919d5e899b259d80d4e"} Jul 17 21:28:11.499223 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.498973 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:28:11.499360 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.499345 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gjgzw" Jul 17 21:28:11.513800 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.513777 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:28:11.523206 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:11.523172 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" podStartSLOduration=9.515287092 podStartE2EDuration="28.52315935s" totalImagesPullingTime="19.007872258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:11.521973715 +0000 UTC m=+28.791971419" watchObservedRunningTime="2026-07-17 21:28:11.52315935 +0000 UTC m=+28.793157031" Jul 17 21:28:12.292339 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:12.292302 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:12.292660 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:12.292302 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:12.292660 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:12.292424 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:12.292660 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:12.292490 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:12.501802 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:12.501776 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:28:12.502199 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:12.501840 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:28:12.516917 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:12.516896 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:28:13.295449 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:13.295416 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:13.295593 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:13.295504 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:13.505179 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:13.505141 2566 generic.go:356] "Generic (PLEG): container finished" podID="302e487b-c564-42d3-90d7-a846a77e2490" containerID="1dea6012ad23b5023d7a012fb9b145b13b62ba3ceca125f27cd06b0c10b556d3" exitCode=0 Jul 17 21:28:13.505557 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:13.505244 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerDied","Data":"1dea6012ad23b5023d7a012fb9b145b13b62ba3ceca125f27cd06b0c10b556d3"} Jul 17 21:28:14.292480 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:14.292447 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:14.292618 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:14.292572 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:14.292699 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:14.292641 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:14.292782 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:14.292764 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:15.292800 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:15.292773 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:15.293174 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:15.292871 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:15.511323 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:15.511290 2566 generic.go:356] "Generic (PLEG): container finished" podID="302e487b-c564-42d3-90d7-a846a77e2490" containerID="c602e92a75e7ae6847383fc52a734fa6baa4740cdce3a08ab8a8e079e9b8176a" exitCode=0 Jul 17 21:28:15.511480 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:15.511338 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerDied","Data":"c602e92a75e7ae6847383fc52a734fa6baa4740cdce3a08ab8a8e079e9b8176a"} Jul 17 21:28:16.292368 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:16.292333 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:16.292552 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:16.292453 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:16.292552 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:16.292522 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:16.292659 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:16.292641 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:16.951770 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:16.951729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:16.951770 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:16.951776 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:16.952280 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:16.951889 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:28:16.952280 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:16.951891 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:28:16.952280 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:16.951950 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs podName:29753f77-c24d-42c7-8518-24fd9698d3ac nodeName:}" failed. No retries permitted until 2026-07-17 21:28:48.951935826 +0000 UTC m=+66.221933512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs") pod "network-metrics-daemon-g4x5v" (UID: "29753f77-c24d-42c7-8518-24fd9698d3ac") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:28:16.952280 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:16.951970 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret podName:4f6911d6-a498-4c63-bb37-156b93e3a189 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:48.951960605 +0000 UTC m=+66.221958289 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret") pod "global-pull-secret-syncer-xbgxj" (UID: "4f6911d6-a498-4c63-bb37-156b93e3a189") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:28:17.052840 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:17.052805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:17.053031 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:17.052984 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:28:17.053031 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:17.053009 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:28:17.053031 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:17.053022 2566 projected.go:196] Error preparing data for projected volume kube-api-access-kswmr for pod openshift-network-diagnostics/network-check-target-467kv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:28:17.053190 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:17.053087 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr podName:b3219454-f5d7-41d3-8ad9-662ae16c50b2 nodeName:}" failed. No retries permitted until 2026-07-17 21:28:49.053068196 +0000 UTC m=+66.323065902 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kswmr" (UniqueName: "kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr") pod "network-check-target-467kv" (UID: "b3219454-f5d7-41d3-8ad9-662ae16c50b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:28:17.293110 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:17.292627 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:17.293110 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:17.292748 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:18.292669 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:18.292630 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:18.293074 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:18.292630 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:18.293074 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:18.292779 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:18.293074 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:18.292798 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:19.292084 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:19.292050 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:19.292280 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:19.292162 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:20.292362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:20.292322 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:20.292362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:20.292352 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:20.292775 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:20.292449 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:20.292775 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:20.292542 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:20.525280 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:20.525000 2566 generic.go:356] "Generic (PLEG): container finished" podID="302e487b-c564-42d3-90d7-a846a77e2490" containerID="83f8b45e67ed1542461761ea0a29c588ad3fa4a2ca6d1e59e5d8e8c70b3d05c4" exitCode=0 Jul 17 21:28:20.525450 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:20.525073 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerDied","Data":"83f8b45e67ed1542461761ea0a29c588ad3fa4a2ca6d1e59e5d8e8c70b3d05c4"} Jul 17 21:28:21.293042 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:21.293002 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:21.293583 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:21.293100 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:21.529986 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:21.529949 2566 generic.go:356] "Generic (PLEG): container finished" podID="302e487b-c564-42d3-90d7-a846a77e2490" containerID="65903a625dbcd11a18579a70ab917819fe181dd908a7a9eb55fcd0318778bd62" exitCode=0 Jul 17 21:28:21.530126 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:21.530004 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerDied","Data":"65903a625dbcd11a18579a70ab917819fe181dd908a7a9eb55fcd0318778bd62"} Jul 17 21:28:22.292348 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:22.292321 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:22.292537 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:22.292432 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:22.292537 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:22.292495 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:22.292646 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:22.292602 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:22.535338 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:22.535310 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" event={"ID":"302e487b-c564-42d3-90d7-a846a77e2490","Type":"ContainerStarted","Data":"9a18b0f17d64361d4a8666c9f74439cb61cf98c864a842d5ab71dfb592110a0e"} Jul 17 21:28:22.559910 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:22.559828 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kfh2f" podStartSLOduration=9.600517771 podStartE2EDuration="39.559812631s" totalImagesPullingTime="29.95929486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:22.559490418 +0000 UTC m=+39.829488122" watchObservedRunningTime="2026-07-17 21:28:22.559812631 +0000 UTC m=+39.829810335" Jul 17 21:28:23.292882 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:23.292844 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:23.293047 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:23.292928 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:24.292512 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:24.292474 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:24.292891 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:24.292493 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:24.292891 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:24.292608 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:24.292891 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:24.292648 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:25.292360 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:25.292323 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:25.292504 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:25.292432 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:26.292959 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:26.292922 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:26.293454 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:26.293052 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:26.293454 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:26.293099 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:26.293454 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:26.293185 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:27.292208 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:27.292175 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:27.292411 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:27.292335 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:27.985518 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:27.985484 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbgxj"] Jul 17 21:28:27.986180 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:27.985596 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:27.986180 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:27.985684 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:27.988904 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:27.988874 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-467kv"] Jul 17 21:28:27.989053 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:27.988929 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g4x5v"] Jul 17 21:28:27.989053 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:27.989004 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:27.989136 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:27.989096 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:27.989293 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:27.989275 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:27.989372 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:27.989354 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:29.293003 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:29.292975 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:29.293439 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:29.293073 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:29.293439 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:29.293084 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:29.293439 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:29.292984 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:29.293439 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:29.293172 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:29.293439 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:29.293217 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:31.292728 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:31.292687 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:31.293180 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:31.292741 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:31.293180 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:31.292683 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:31.293180 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:31.292807 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4x5v" podUID="29753f77-c24d-42c7-8518-24fd9698d3ac" Jul 17 21:28:31.293180 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:31.292934 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbgxj" podUID="4f6911d6-a498-4c63-bb37-156b93e3a189" Jul 17 21:28:31.293180 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:31.292984 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-467kv" podUID="b3219454-f5d7-41d3-8ad9-662ae16c50b2" Jul 17 21:28:32.034049 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.032968 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-178.ec2.internal" event="NodeReady" Jul 17 21:28:32.034049 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.033987 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 21:28:32.089805 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.089717 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-df5k2"] Jul 17 21:28:32.110570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.110535 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h2pz6"] Jul 17 21:28:32.110738 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.110652 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.113159 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.113132 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 21:28:32.113337 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.113219 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qt4k4\"" Jul 17 21:28:32.113459 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.113447 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 21:28:32.129096 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.129070 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h2pz6"] Jul 17 21:28:32.129096 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.129100 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-df5k2"] Jul 17 21:28:32.129338 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.129164 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.131537 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.131519 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 21:28:32.131645 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.131592 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 21:28:32.131816 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.131788 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dhbkk\"" Jul 17 21:28:32.131907 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.131822 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 21:28:32.261273 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.261210 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmtf\" (UniqueName: \"kubernetes.io/projected/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-kube-api-access-9wmtf\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.261273 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.261275 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4aa80dd6-b323-48a1-b800-36603b5acaca-cert\") pod \"ingress-canary-h2pz6\" (UID: \"4aa80dd6-b323-48a1-b800-36603b5acaca\") " pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.261502 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.261319 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-tmp-dir\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.261502 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.261353 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-metrics-tls\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.261502 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.261422 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-config-volume\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.261502 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.261449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqnb\" (UniqueName: \"kubernetes.io/projected/4aa80dd6-b323-48a1-b800-36603b5acaca-kube-api-access-gkqnb\") pod \"ingress-canary-h2pz6\" (UID: \"4aa80dd6-b323-48a1-b800-36603b5acaca\") " pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.362765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.362651 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-config-volume\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.362765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.362690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqnb\" (UniqueName: \"kubernetes.io/projected/4aa80dd6-b323-48a1-b800-36603b5acaca-kube-api-access-gkqnb\") pod \"ingress-canary-h2pz6\" (UID: \"4aa80dd6-b323-48a1-b800-36603b5acaca\") " pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.362765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.362745 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmtf\" (UniqueName: \"kubernetes.io/projected/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-kube-api-access-9wmtf\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.363289 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.362774 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4aa80dd6-b323-48a1-b800-36603b5acaca-cert\") pod \"ingress-canary-h2pz6\" (UID: \"4aa80dd6-b323-48a1-b800-36603b5acaca\") " pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.363289 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.362804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-tmp-dir\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.363289 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.362841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-metrics-tls\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.363289 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.363200 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-tmp-dir\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.363472 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.363369 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-config-volume\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.367374 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.367352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4aa80dd6-b323-48a1-b800-36603b5acaca-cert\") pod \"ingress-canary-h2pz6\" (UID: \"4aa80dd6-b323-48a1-b800-36603b5acaca\") " pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.371814 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.371787 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmtf\" (UniqueName: \"kubernetes.io/projected/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-kube-api-access-9wmtf\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.372075 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.372059 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqnb\" (UniqueName: \"kubernetes.io/projected/4aa80dd6-b323-48a1-b800-36603b5acaca-kube-api-access-gkqnb\") pod \"ingress-canary-h2pz6\" (UID: \"4aa80dd6-b323-48a1-b800-36603b5acaca\") " pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.375121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.375103 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2a2ed7-2f27-42ea-9516-f224e975dcd2-metrics-tls\") pod \"dns-default-df5k2\" (UID: \"ad2a2ed7-2f27-42ea-9516-f224e975dcd2\") " pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.420462 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.420420 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:32.437575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.437543 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h2pz6" Jul 17 21:28:32.599643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.598559 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h2pz6"] Jul 17 21:28:32.601328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:32.601302 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-df5k2"] Jul 17 21:28:32.605443 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:32.604417 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa80dd6_b323_48a1_b800_36603b5acaca.slice/crio-fae0d6dc766ec9a5c838d215e9f618464389f586c0281a2a3f3f1cd3c2c90c5b WatchSource:0}: Error finding container fae0d6dc766ec9a5c838d215e9f618464389f586c0281a2a3f3f1cd3c2c90c5b: Status 404 returned error can't find the container with id fae0d6dc766ec9a5c838d215e9f618464389f586c0281a2a3f3f1cd3c2c90c5b Jul 17 21:28:32.607684 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:32.607658 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2a2ed7_2f27_42ea_9516_f224e975dcd2.slice/crio-6ce82bbfd93a64c910f0864a8ed82642c420417023f76b25a10c9ada1e4d055f WatchSource:0}: Error finding container 6ce82bbfd93a64c910f0864a8ed82642c420417023f76b25a10c9ada1e4d055f: Status 404 returned error can't find the container with id 6ce82bbfd93a64c910f0864a8ed82642c420417023f76b25a10c9ada1e4d055f Jul 17 21:28:33.185174 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.185141 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54fc5687d-vjvjr"] Jul 17 21:28:33.198378 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.198223 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fc5687d-vjvjr"] Jul 17 21:28:33.198378 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.198373 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.201617 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.201404 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 21:28:33.201617 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.201497 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 21:28:33.202509 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.202483 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 21:28:33.202509 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.202501 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 21:28:33.203318 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.203133 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 21:28:33.203572 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.203521 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cnsv4\"" Jul 17 21:28:33.203572 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.203539 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 21:28:33.203939 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.203777 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 21:28:33.208472 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.208446 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 21:28:33.270172 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.270132 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-oauth-config\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.270172 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.270175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-config\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.270431 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.270215 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-service-ca\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.270431 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.270259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-oauth-serving-cert\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.270431 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.270305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpltz\" (UniqueName: \"kubernetes.io/projected/044d02a6-5527-4cb2-bcfa-90632edbb4a5-kube-api-access-tpltz\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.270431 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.270338 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-trusted-ca-bundle\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.270431 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.270363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-serving-cert\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.297360 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.297332 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:33.297684 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.297670 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:33.298016 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.297995 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:33.301204 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.300696 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:28:33.301204 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.300916 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7k8fl\"" Jul 17 21:28:33.301204 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.300923 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:28:33.301204 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.301050 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:28:33.301204 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.301074 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:28:33.301533 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.301262 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fkx7\"" Jul 17 21:28:33.370705 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.370664 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-oauth-config\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.370711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-config\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.370987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-service-ca\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.371019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-oauth-serving-cert\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.371054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpltz\" (UniqueName: \"kubernetes.io/projected/044d02a6-5527-4cb2-bcfa-90632edbb4a5-kube-api-access-tpltz\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.371085 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-trusted-ca-bundle\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.371103 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-serving-cert\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371580 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.371531 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-config\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.371886 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.371827 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-service-ca\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.372151 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.372103 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-oauth-serving-cert\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.372447 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.372362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-trusted-ca-bundle\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.374243 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.374198 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-serving-cert\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.374326 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.374306 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-oauth-config\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.383198 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.383170 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpltz\" (UniqueName: \"kubernetes.io/projected/044d02a6-5527-4cb2-bcfa-90632edbb4a5-kube-api-access-tpltz\") pod \"console-54fc5687d-vjvjr\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.513908 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.513828 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:33.565430 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.565306 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h2pz6" event={"ID":"4aa80dd6-b323-48a1-b800-36603b5acaca","Type":"ContainerStarted","Data":"fae0d6dc766ec9a5c838d215e9f618464389f586c0281a2a3f3f1cd3c2c90c5b"} Jul 17 21:28:33.567207 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.567144 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-df5k2" event={"ID":"ad2a2ed7-2f27-42ea-9516-f224e975dcd2","Type":"ContainerStarted","Data":"6ce82bbfd93a64c910f0864a8ed82642c420417023f76b25a10c9ada1e4d055f"} Jul 17 21:28:33.702330 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:33.702035 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fc5687d-vjvjr"] Jul 17 21:28:33.706273 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:33.706223 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044d02a6_5527_4cb2_bcfa_90632edbb4a5.slice/crio-614e0d6256d4e3c54c2b80cf33e11b42324df17d2bed89b0061722d579e5a634 WatchSource:0}: Error finding container 614e0d6256d4e3c54c2b80cf33e11b42324df17d2bed89b0061722d579e5a634: Status 404 returned error can't find the container with id 614e0d6256d4e3c54c2b80cf33e11b42324df17d2bed89b0061722d579e5a634 Jul 17 21:28:34.571688 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:34.571636 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fc5687d-vjvjr" event={"ID":"044d02a6-5527-4cb2-bcfa-90632edbb4a5","Type":"ContainerStarted","Data":"614e0d6256d4e3c54c2b80cf33e11b42324df17d2bed89b0061722d579e5a634"} Jul 17 21:28:36.580292 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:36.580137 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-df5k2" event={"ID":"ad2a2ed7-2f27-42ea-9516-f224e975dcd2","Type":"ContainerStarted","Data":"e1804468f19d88e345100d9e84198d60ae54e5b19c209ee768518524832775b6"} Jul 17 21:28:36.580292 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:36.580210 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:36.580292 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:36.580255 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-df5k2" event={"ID":"ad2a2ed7-2f27-42ea-9516-f224e975dcd2","Type":"ContainerStarted","Data":"3c3127d0f9791e675df9e6bc2d6dacf46c6c2f7d8a701348b8f4a44f6a4e6bed"} Jul 17 21:28:36.581973 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:36.581757 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h2pz6" event={"ID":"4aa80dd6-b323-48a1-b800-36603b5acaca","Type":"ContainerStarted","Data":"dc56ea5c5b4866beab8531c058f476d3a27c9c301ba537120e0f07ec8655887f"} Jul 17 21:28:36.598720 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:36.598667 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-df5k2" podStartSLOduration=1.472602025 podStartE2EDuration="4.598652767s" totalImagesPullingTime="3.126050742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:36.597735828 +0000 UTC m=+53.867733532" watchObservedRunningTime="2026-07-17 21:28:36.598652767 +0000 UTC m=+53.868650462" Jul 17 21:28:36.614601 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:36.614548 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h2pz6" podStartSLOduration=1.477612247 podStartE2EDuration="4.614531072s" totalImagesPullingTime="3.136918825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:36.614294314 +0000 UTC m=+53.884292039" watchObservedRunningTime="2026-07-17 21:28:36.614531072 +0000 UTC m=+53.884528775" Jul 17 21:28:37.585640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:37.585605 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fc5687d-vjvjr" event={"ID":"044d02a6-5527-4cb2-bcfa-90632edbb4a5","Type":"ContainerStarted","Data":"3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276"} Jul 17 21:28:39.967268 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.967194 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54fc5687d-vjvjr" podStartSLOduration=3.302210186 podStartE2EDuration="6.967176525s" totalImagesPullingTime="3.664966339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:37.605820179 +0000 UTC m=+54.875817900" watchObservedRunningTime="2026-07-17 21:28:39.967176525 +0000 UTC m=+57.237174207" Jul 17 21:28:39.967713 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.967361 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7hk"] Jul 17 21:28:39.970130 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.970114 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:39.972544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.972519 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 21:28:39.972795 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.972776 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 21:28:39.972897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.972796 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jn2hx\"" Jul 17 21:28:39.973558 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.973542 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 21:28:39.973647 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.973629 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 21:28:39.985250 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:39.985204 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7hk"] Jul 17 21:28:40.023384 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.023349 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfkw\" (UniqueName: \"kubernetes.io/projected/00f31afe-10c6-46c9-bed8-79a188a026da-kube-api-access-7gfkw\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.023384 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.023385 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00f31afe-10c6-46c9-bed8-79a188a026da-crio-socket\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.023620 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.023457 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00f31afe-10c6-46c9-bed8-79a188a026da-data-volume\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.023620 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.023489 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.023620 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.023543 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.076860 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.076821 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8"] Jul 17 21:28:40.078681 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.078660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" Jul 17 21:28:40.081285 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.081264 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 17 21:28:40.081367 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.081313 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kkpqp\"" Jul 17 21:28:40.091619 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.091596 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8"] Jul 17 21:28:40.124089 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfkw\" (UniqueName: \"kubernetes.io/projected/00f31afe-10c6-46c9-bed8-79a188a026da-kube-api-access-7gfkw\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.124089 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124093 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00f31afe-10c6-46c9-bed8-79a188a026da-crio-socket\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.124360 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00f31afe-10c6-46c9-bed8-79a188a026da-data-volume\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.124360 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124309 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.124456 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.124456 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124392 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0a4ceea4-d7ad-428d-b9f9-ea2fe6155094-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-lxrb8\" (UID: \"0a4ceea4-d7ad-428d-b9f9-ea2fe6155094\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" Jul 17 21:28:40.124456 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.124421 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:40.124456 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124445 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00f31afe-10c6-46c9-bed8-79a188a026da-crio-socket\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.124601 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.124473 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00f31afe-10c6-46c9-bed8-79a188a026da-data-volume\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.124601 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.124478 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:40.124601 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.124486 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:40.624471349 +0000 UTC m=+57.894469031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : configmap "kube-rbac-proxy" not found Jul 17 21:28:40.124601 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.124547 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:40.624534444 +0000 UTC m=+57.894532130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:40.135351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.135326 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfkw\" (UniqueName: \"kubernetes.io/projected/00f31afe-10c6-46c9-bed8-79a188a026da-kube-api-access-7gfkw\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.225118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.225025 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0a4ceea4-d7ad-428d-b9f9-ea2fe6155094-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-lxrb8\" (UID: \"0a4ceea4-d7ad-428d-b9f9-ea2fe6155094\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" Jul 17 21:28:40.227512 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.227494 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0a4ceea4-d7ad-428d-b9f9-ea2fe6155094-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-lxrb8\" (UID: \"0a4ceea4-d7ad-428d-b9f9-ea2fe6155094\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" Jul 17 21:28:40.387656 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.387616 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" Jul 17 21:28:40.512583 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.512508 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8"] Jul 17 21:28:40.515471 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:40.515448 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4ceea4_d7ad_428d_b9f9_ea2fe6155094.slice/crio-319d4e85a7d9fab9ba2ebc37214501b3c0d6adc4d274002016502ad91aa2d6c8 WatchSource:0}: Error finding container 319d4e85a7d9fab9ba2ebc37214501b3c0d6adc4d274002016502ad91aa2d6c8: Status 404 returned error can't find the container with id 319d4e85a7d9fab9ba2ebc37214501b3c0d6adc4d274002016502ad91aa2d6c8 Jul 17 21:28:40.594958 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.594918 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" event={"ID":"0a4ceea4-d7ad-428d-b9f9-ea2fe6155094","Type":"ContainerStarted","Data":"319d4e85a7d9fab9ba2ebc37214501b3c0d6adc4d274002016502ad91aa2d6c8"} Jul 17 21:28:40.628387 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.628360 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.628543 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:40.628439 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:40.628543 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.628509 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:40.628543 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.628519 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:40.628656 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.628560 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:41.628546421 +0000 UTC m=+58.898544103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : configmap "kube-rbac-proxy" not found Jul 17 21:28:40.628656 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:40.628580 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:41.628565707 +0000 UTC m=+58.898563389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:41.598793 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:41.598761 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" event={"ID":"0a4ceea4-d7ad-428d-b9f9-ea2fe6155094","Type":"ContainerStarted","Data":"743ef061ea07140237478adcafe0c851bc178aea1d050721f91adeefc2e6493c"} Jul 17 21:28:41.599187 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:41.598921 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" Jul 17 21:28:41.603420 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:41.603396 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" Jul 17 21:28:41.614123 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:41.614082 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-lxrb8" podStartSLOduration=0.624127452 podStartE2EDuration="1.614066666s" totalImagesPullingTime="989.939214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:41.613321063 +0000 UTC m=+58.883318768" watchObservedRunningTime="2026-07-17 21:28:41.614066666 +0000 UTC m=+58.884064370" Jul 17 21:28:41.636708 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:41.636678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:41.636867 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:41.636733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:41.636867 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:41.636820 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:41.636972 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:41.636887 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:43.636870766 +0000 UTC m=+60.906868448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : configmap "kube-rbac-proxy" not found Jul 17 21:28:41.636972 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:41.636887 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:41.636972 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:41.636932 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:43.636920461 +0000 UTC m=+60.906918142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:42.147558 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.147521 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k"] Jul 17 21:28:42.149365 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.149349 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.154469 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.154444 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 21:28:42.155822 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.155513 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 21:28:42.155822 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.155570 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wwnkv\"" Jul 17 21:28:42.155822 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.155521 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 21:28:42.155822 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.155697 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 21:28:42.155822 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.155812 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 21:28:42.159398 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.159380 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k"] Jul 17 21:28:42.241831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.241786 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/749c3879-6357-483a-8640-795e87d62b2c-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.242010 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.241860 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zb9k\" (UniqueName: \"kubernetes.io/projected/749c3879-6357-483a-8640-795e87d62b2c-kube-api-access-2zb9k\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.242010 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.241920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/749c3879-6357-483a-8640-795e87d62b2c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.242010 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.241949 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/749c3879-6357-483a-8640-795e87d62b2c-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.342737 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.342698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zb9k\" (UniqueName: \"kubernetes.io/projected/749c3879-6357-483a-8640-795e87d62b2c-kube-api-access-2zb9k\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.342910 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.342770 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/749c3879-6357-483a-8640-795e87d62b2c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.342910 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.342790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/749c3879-6357-483a-8640-795e87d62b2c-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.342910 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.342821 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/749c3879-6357-483a-8640-795e87d62b2c-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.343486 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.343457 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/749c3879-6357-483a-8640-795e87d62b2c-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.345295 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.345274 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/749c3879-6357-483a-8640-795e87d62b2c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.345385 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.345356 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/749c3879-6357-483a-8640-795e87d62b2c-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.352140 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.352118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zb9k\" (UniqueName: \"kubernetes.io/projected/749c3879-6357-483a-8640-795e87d62b2c-kube-api-access-2zb9k\") pod \"prometheus-operator-5945b5cfd-c2l8k\" (UID: \"749c3879-6357-483a-8640-795e87d62b2c\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.365934 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.365910 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54fc5687d-vjvjr_044d02a6-5527-4cb2-bcfa-90632edbb4a5/console/0.log" Jul 17 21:28:42.458332 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.458302 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" Jul 17 21:28:42.587030 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.586995 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k"] Jul 17 21:28:42.590093 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:42.590060 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749c3879_6357_483a_8640_795e87d62b2c.slice/crio-5c407404f957f28c66e440dac6fc117590a699ab5976aa56349698eb1b3e0a66 WatchSource:0}: Error finding container 5c407404f957f28c66e440dac6fc117590a699ab5976aa56349698eb1b3e0a66: Status 404 returned error can't find the container with id 5c407404f957f28c66e440dac6fc117590a699ab5976aa56349698eb1b3e0a66 Jul 17 21:28:42.601921 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:42.601897 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" event={"ID":"749c3879-6357-483a-8640-795e87d62b2c","Type":"ContainerStarted","Data":"5c407404f957f28c66e440dac6fc117590a699ab5976aa56349698eb1b3e0a66"} Jul 17 21:28:43.350501 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.350474 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-df5k2_ad2a2ed7-2f27-42ea-9516-f224e975dcd2/dns/0.log" Jul 17 21:28:43.514824 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.514789 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:43.515032 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.514855 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:43.520308 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.520278 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:43.550004 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.549970 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-df5k2_ad2a2ed7-2f27-42ea-9516-f224e975dcd2/kube-rbac-proxy/0.log" Jul 17 21:28:43.609162 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.609080 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:28:43.653121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.652690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:43.653121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:43.652749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:43.653121 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:43.652958 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:43.653121 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:43.653017 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:43.653121 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:43.653021 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.653000146 +0000 UTC m=+64.922997832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : configmap "kube-rbac-proxy" not found Jul 17 21:28:43.653121 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:43.653086 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.653068551 +0000 UTC m=+64.923066239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:44.351843 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:44.351811 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5zrcf_1c44a901-fbcb-40f0-9984-123b95d7dbb5/dns-node-resolver/0.log" Jul 17 21:28:44.526243 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:44.526200 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phlpq" Jul 17 21:28:44.610195 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:44.610107 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" event={"ID":"749c3879-6357-483a-8640-795e87d62b2c","Type":"ContainerStarted","Data":"b94a0fdbb101d09345e740ea02d4c231a4decb69a97cda0c5d402041cabebc81"} Jul 17 21:28:44.610195 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:44.610161 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" event={"ID":"749c3879-6357-483a-8640-795e87d62b2c","Type":"ContainerStarted","Data":"7a85457a1c3226a8fe9aef41646d5dba841d7659e4c32bf8449757841bea99fc"} Jul 17 21:28:44.628000 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:44.627956 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c2l8k" podStartSLOduration=1.433816205 podStartE2EDuration="2.627936403s" totalImagesPullingTime="1.194120198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:44.627648895 +0000 UTC m=+61.897646598" watchObservedRunningTime="2026-07-17 21:28:44.627936403 +0000 UTC m=+61.897934108" Jul 17 21:28:45.150334 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:45.150310 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-52cmr_45e3a6c9-bb86-424c-8909-feafef42373b/node-ca/0.log" Jul 17 21:28:45.949525 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:45.949488 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h2pz6_4aa80dd6-b323-48a1-b800-36603b5acaca/serve-healthcheck-canary/0.log" Jul 17 21:28:46.496134 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.496095 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-knzmh"] Jul 17 21:28:46.498933 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.498917 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.501892 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.501847 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 21:28:46.502059 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.502038 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vtfpb\"" Jul 17 21:28:46.502121 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.502039 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 21:28:46.503052 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.503034 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 21:28:46.516932 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.516902 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-knzmh"] Jul 17 21:28:46.520670 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.520648 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-72t6s"] Jul 17 21:28:46.522636 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.522623 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.525510 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.525487 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 21:28:46.525618 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.525546 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 21:28:46.525618 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.525563 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 21:28:46.525618 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.525546 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cbqw8\"" Jul 17 21:28:46.577825 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.577789 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-accelerators-collector-config\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.577825 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.577825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxqw\" (UniqueName: \"kubernetes.io/projected/144d3e2b-9352-47e1-bb25-434bb2266554-kube-api-access-wqxqw\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.578083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.577845 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.578083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.577866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.578083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.577962 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-tls\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.578083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.577995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.578083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.578083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578079 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-wtmp\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.578343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578118 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-sys\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.578343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578135 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dz7\" (UniqueName: \"kubernetes.io/projected/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-api-access-p7dz7\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.578343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-textfile\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.578343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578166 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.578343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578223 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.578343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d3e2b-9352-47e1-bb25-434bb2266554-metrics-client-ca\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.578343 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.578325 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-root\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.587573 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.587552 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-df5k2" Jul 17 21:28:46.679321 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-accelerators-collector-config\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679321 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxqw\" (UniqueName: \"kubernetes.io/projected/144d3e2b-9352-47e1-bb25-434bb2266554-kube-api-access-wqxqw\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679341 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.679570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679415 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-tls\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.679570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679478 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.679570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679508 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-wtmp\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679545 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-sys\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dz7\" (UniqueName: \"kubernetes.io/projected/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-api-access-p7dz7\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679604 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-textfile\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:46.679607 2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:46.679688 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-tls podName:1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d nodeName:}" failed. No retries permitted until 2026-07-17 21:28:47.179665519 +0000 UTC m=+64.449663217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-knzmh" (UID: "1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d") : secret "kube-state-metrics-tls" not found Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679721 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-sys\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.679918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.680362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679996 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-wtmp\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.680362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.679997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d3e2b-9352-47e1-bb25-434bb2266554-metrics-client-ca\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.680362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.680045 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-root\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.680362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.680129 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/144d3e2b-9352-47e1-bb25-434bb2266554-root\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.680362 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.680137 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-textfile\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.680623 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.680598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144d3e2b-9352-47e1-bb25-434bb2266554-metrics-client-ca\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.680694 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.680668 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.680853 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.680833 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.680957 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.680935 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-accelerators-collector-config\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.682191 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.682172 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-tls\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.682345 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.682328 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.682389 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.682333 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144d3e2b-9352-47e1-bb25-434bb2266554-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.690518 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.690497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dz7\" (UniqueName: \"kubernetes.io/projected/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-api-access-p7dz7\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:46.690606 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.690498 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxqw\" (UniqueName: \"kubernetes.io/projected/144d3e2b-9352-47e1-bb25-434bb2266554-kube-api-access-wqxqw\") pod \"node-exporter-72t6s\" (UID: \"144d3e2b-9352-47e1-bb25-434bb2266554\") " pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.831638 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:46.831547 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-72t6s" Jul 17 21:28:46.841249 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:46.841199 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144d3e2b_9352_47e1_bb25_434bb2266554.slice/crio-b7e98f501457de7e451ba474bb19029a717300568656c8cfd6f0f8d3e998ec8e WatchSource:0}: Error finding container b7e98f501457de7e451ba474bb19029a717300568656c8cfd6f0f8d3e998ec8e: Status 404 returned error can't find the container with id b7e98f501457de7e451ba474bb19029a717300568656c8cfd6f0f8d3e998ec8e Jul 17 21:28:47.183485 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.183451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:47.186195 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.186172 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-knzmh\" (UID: \"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:47.407457 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.407423 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" Jul 17 21:28:47.564964 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.564818 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-knzmh"] Jul 17 21:28:47.567325 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:47.567288 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8d31bd_ebd4_4f0a_a394_b6e22fba3e4d.slice/crio-a4af791ce2020a1e102a32943fdff1ec2952d7cccf98d899a4bb7b5c06ed0810 WatchSource:0}: Error finding container a4af791ce2020a1e102a32943fdff1ec2952d7cccf98d899a4bb7b5c06ed0810: Status 404 returned error can't find the container with id a4af791ce2020a1e102a32943fdff1ec2952d7cccf98d899a4bb7b5c06ed0810 Jul 17 21:28:47.621881 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.621843 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72t6s" event={"ID":"144d3e2b-9352-47e1-bb25-434bb2266554","Type":"ContainerStarted","Data":"b7e98f501457de7e451ba474bb19029a717300568656c8cfd6f0f8d3e998ec8e"} Jul 17 21:28:47.623135 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.623106 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" event={"ID":"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d","Type":"ContainerStarted","Data":"a4af791ce2020a1e102a32943fdff1ec2952d7cccf98d899a4bb7b5c06ed0810"} Jul 17 21:28:47.688512 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.688472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:47.688691 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:47.688600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:47.688691 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:47.688650 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 21:28:47.688801 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:47.688710 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 21:28:47.688801 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:47.688727 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:55.688704828 +0000 UTC m=+72.958702512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : secret "insights-runtime-extractor-tls" not found Jul 17 21:28:47.688801 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:28:47.688764 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm podName:00f31afe-10c6-46c9-bed8-79a188a026da nodeName:}" failed. No retries permitted until 2026-07-17 21:28:55.688747555 +0000 UTC m=+72.958745272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7hk" (UID: "00f31afe-10c6-46c9-bed8-79a188a026da") : configmap "kube-rbac-proxy" not found Jul 17 21:28:48.632072 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:48.632039 2566 generic.go:356] "Generic (PLEG): container finished" podID="144d3e2b-9352-47e1-bb25-434bb2266554" containerID="0dba462473d900ed2227ea2cc545ed5e1ac3ba7c95bce42c0196fa5db1b4c89b" exitCode=0 Jul 17 21:28:48.632521 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:48.632117 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72t6s" event={"ID":"144d3e2b-9352-47e1-bb25-434bb2266554","Type":"ContainerDied","Data":"0dba462473d900ed2227ea2cc545ed5e1ac3ba7c95bce42c0196fa5db1b4c89b"} Jul 17 21:28:49.002195 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.002158 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:49.002412 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.002311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:49.005139 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.005110 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:28:49.005139 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.005130 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:28:49.015168 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.015142 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29753f77-c24d-42c7-8518-24fd9698d3ac-metrics-certs\") pod \"network-metrics-daemon-g4x5v\" (UID: \"29753f77-c24d-42c7-8518-24fd9698d3ac\") " pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:49.015623 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.015601 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f6911d6-a498-4c63-bb37-156b93e3a189-original-pull-secret\") pod \"global-pull-secret-syncer-xbgxj\" (UID: \"4f6911d6-a498-4c63-bb37-156b93e3a189\") " pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:49.103348 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.103315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:49.106201 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.106179 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:28:49.116651 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.116628 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:28:49.127342 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.127311 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kswmr\" (UniqueName: \"kubernetes.io/projected/b3219454-f5d7-41d3-8ad9-662ae16c50b2-kube-api-access-kswmr\") pod \"network-check-target-467kv\" (UID: \"b3219454-f5d7-41d3-8ad9-662ae16c50b2\") " pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:49.209987 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.209955 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbgxj" Jul 17 21:28:49.220424 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.220402 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5fkx7\"" Jul 17 21:28:49.227865 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.227848 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7k8fl\"" Jul 17 21:28:49.227980 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.227963 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4x5v" Jul 17 21:28:49.235887 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.235859 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:49.354577 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.354545 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbgxj"] Jul 17 21:28:49.361058 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:49.360494 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6911d6_a498_4c63_bb37_156b93e3a189.slice/crio-6f0c8badc8d0075fb0c78c168a6b8222f7bbdb8f1011d6145a0d1040f30a3255 WatchSource:0}: Error finding container 6f0c8badc8d0075fb0c78c168a6b8222f7bbdb8f1011d6145a0d1040f30a3255: Status 404 returned error can't find the container with id 6f0c8badc8d0075fb0c78c168a6b8222f7bbdb8f1011d6145a0d1040f30a3255 Jul 17 21:28:49.377891 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.377851 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g4x5v"] Jul 17 21:28:49.381067 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:49.381035 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29753f77_c24d_42c7_8518_24fd9698d3ac.slice/crio-35bc5560fed7aaa33ee83a894dbf7ed04e872e063df184cf3a2c41c1a6de2b02 WatchSource:0}: Error finding container 35bc5560fed7aaa33ee83a894dbf7ed04e872e063df184cf3a2c41c1a6de2b02: Status 404 returned error can't find the container with id 35bc5560fed7aaa33ee83a894dbf7ed04e872e063df184cf3a2c41c1a6de2b02 Jul 17 21:28:49.393975 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.393948 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-467kv"] Jul 17 21:28:49.396984 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:49.396960 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3219454_f5d7_41d3_8ad9_662ae16c50b2.slice/crio-50d929fc1be145c5c2f635c3ed4f9872fa334385dea1a5254cd4f7d08e6c7baa WatchSource:0}: Error finding container 50d929fc1be145c5c2f635c3ed4f9872fa334385dea1a5254cd4f7d08e6c7baa: Status 404 returned error can't find the container with id 50d929fc1be145c5c2f635c3ed4f9872fa334385dea1a5254cd4f7d08e6c7baa Jul 17 21:28:49.637272 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.637167 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72t6s" event={"ID":"144d3e2b-9352-47e1-bb25-434bb2266554","Type":"ContainerStarted","Data":"67959ea2e1a0482b3f816968a0e4d6f0fd7dc96214a85faf10eecb3cc0f729a3"} Jul 17 21:28:49.637272 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.637246 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72t6s" event={"ID":"144d3e2b-9352-47e1-bb25-434bb2266554","Type":"ContainerStarted","Data":"ceb97e2b551a294399336c477bc7e6a692f6513746739741626c67ae13c15895"} Jul 17 21:28:49.638463 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.638441 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbgxj" event={"ID":"4f6911d6-a498-4c63-bb37-156b93e3a189","Type":"ContainerStarted","Data":"6f0c8badc8d0075fb0c78c168a6b8222f7bbdb8f1011d6145a0d1040f30a3255"} Jul 17 21:28:49.640455 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.640423 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" event={"ID":"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d","Type":"ContainerStarted","Data":"3e472d5237b9cc65fc5dd161df8bb710d29ce05da8f8796e2450601f6f504d15"} Jul 17 21:28:49.640527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.640478 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" event={"ID":"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d","Type":"ContainerStarted","Data":"103a965c11d896d8151d31df3ec759993b49cc15d0a055ad5ff7be112bd07169"} Jul 17 21:28:49.640527 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.640509 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" event={"ID":"1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d","Type":"ContainerStarted","Data":"da8197829d89be7ff3827f8cc307a49ce07dfda26c1d98b0173b9251cb05b3dc"} Jul 17 21:28:49.641560 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.641534 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-467kv" event={"ID":"b3219454-f5d7-41d3-8ad9-662ae16c50b2","Type":"ContainerStarted","Data":"50d929fc1be145c5c2f635c3ed4f9872fa334385dea1a5254cd4f7d08e6c7baa"} Jul 17 21:28:49.642599 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.642581 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g4x5v" event={"ID":"29753f77-c24d-42c7-8518-24fd9698d3ac","Type":"ContainerStarted","Data":"35bc5560fed7aaa33ee83a894dbf7ed04e872e063df184cf3a2c41c1a6de2b02"} Jul 17 21:28:49.654985 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.654942 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-72t6s" podStartSLOduration=2.53705503 podStartE2EDuration="3.654927084s" totalImagesPullingTime="1.046038149s" totalInitContainerRuntime="71.833905ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:49.654000003 +0000 UTC m=+66.923997720" watchObservedRunningTime="2026-07-17 21:28:49.654927084 +0000 UTC m=+66.924924788" Jul 17 21:28:49.670957 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:49.670914 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-knzmh" podStartSLOduration=2.550732544 podStartE2EDuration="3.670900382s" totalImagesPullingTime="1.120167838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:49.670194848 +0000 UTC m=+66.940192552" watchObservedRunningTime="2026-07-17 21:28:49.670900382 +0000 UTC m=+66.940898085" Jul 17 21:28:50.799022 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.798986 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-589fb9d4f5-hcd2p"] Jul 17 21:28:50.803549 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.803523 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:50.806575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.806539 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fk8gv\"" Jul 17 21:28:50.806817 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.806799 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 21:28:50.807634 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.807577 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5ggkf5omr8bbf\"" Jul 17 21:28:50.807973 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.807778 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 21:28:50.807973 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.807967 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 21:28:50.808422 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.808146 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 21:28:50.812986 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.812947 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-589fb9d4f5-hcd2p"] Jul 17 21:28:50.920683 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.920475 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-secret-metrics-server-client-certs\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:50.920683 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.920528 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-secret-metrics-server-tls\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:50.920683 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.920619 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf151ce-e42d-47f3-a732-d982b179c7e7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:50.921115 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.920705 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlkhl\" (UniqueName: \"kubernetes.io/projected/bcf151ce-e42d-47f3-a732-d982b179c7e7-kube-api-access-mlkhl\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:50.921115 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.920735 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcf151ce-e42d-47f3-a732-d982b179c7e7-metrics-server-audit-profiles\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:50.921115 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.920781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-client-ca-bundle\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:50.921115 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:50.920798 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcf151ce-e42d-47f3-a732-d982b179c7e7-audit-log\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.021833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-secret-metrics-server-client-certs\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.021885 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-secret-metrics-server-tls\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.021926 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf151ce-e42d-47f3-a732-d982b179c7e7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.021983 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlkhl\" (UniqueName: \"kubernetes.io/projected/bcf151ce-e42d-47f3-a732-d982b179c7e7-kube-api-access-mlkhl\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.022016 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcf151ce-e42d-47f3-a732-d982b179c7e7-metrics-server-audit-profiles\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.022083 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-client-ca-bundle\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.022126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcf151ce-e42d-47f3-a732-d982b179c7e7-audit-log\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.022610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.022572 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bcf151ce-e42d-47f3-a732-d982b179c7e7-audit-log\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.023377 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.023166 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bcf151ce-e42d-47f3-a732-d982b179c7e7-metrics-server-audit-profiles\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.023566 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.023520 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bcf151ce-e42d-47f3-a732-d982b179c7e7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.024939 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.024914 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-secret-metrics-server-client-certs\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.025804 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.025778 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-secret-metrics-server-tls\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.026930 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.026904 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bcf151ce-e42d-47f3-a732-d982b179c7e7-client-ca-bundle\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.031460 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.031437 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlkhl\" (UniqueName: \"kubernetes.io/projected/bcf151ce-e42d-47f3-a732-d982b179c7e7-kube-api-access-mlkhl\") pod \"metrics-server-589fb9d4f5-hcd2p\" (UID: \"bcf151ce-e42d-47f3-a732-d982b179c7e7\") " pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.120798 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.120762 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:28:51.284709 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.284668 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-589fb9d4f5-hcd2p"] Jul 17 21:28:51.290547 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:51.290519 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf151ce_e42d_47f3_a732_d982b179c7e7.slice/crio-7e06b462e038d2215c5dd1764b6a63d55f1a1a5df589b9c0b8c735b77b231793 WatchSource:0}: Error finding container 7e06b462e038d2215c5dd1764b6a63d55f1a1a5df589b9c0b8c735b77b231793: Status 404 returned error can't find the container with id 7e06b462e038d2215c5dd1764b6a63d55f1a1a5df589b9c0b8c735b77b231793 Jul 17 21:28:51.290899 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.290871 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6"] Jul 17 21:28:51.324032 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.324000 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55b6fc788c-gvj9k"] Jul 17 21:28:51.324391 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.324364 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.327209 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.327178 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 21:28:51.327351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.327186 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mj2jk\"" Jul 17 21:28:51.338928 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.338313 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6"] Jul 17 21:28:51.338928 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.338437 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.347118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.347087 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b6fc788c-gvj9k"] Jul 17 21:28:51.425384 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425348 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-service-ca\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.425553 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-trusted-ca-bundle\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.425553 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425468 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd61feb2-effd-4c91-a9af-2c04eea7a815-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-qzsz6\" (UID: \"cd61feb2-effd-4c91-a9af-2c04eea7a815\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.425553 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425516 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl78p\" (UniqueName: \"kubernetes.io/projected/cd61feb2-effd-4c91-a9af-2c04eea7a815-kube-api-access-cl78p\") pod \"monitoring-plugin-7b7b7bc8bf-qzsz6\" (UID: \"cd61feb2-effd-4c91-a9af-2c04eea7a815\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.425724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425569 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-serving-cert\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.425724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425588 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-oauth-serving-cert\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.425724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-oauth-config\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.425724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-config\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.425724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.425668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfws6\" (UniqueName: \"kubernetes.io/projected/4e167460-3859-4a32-b1e8-8a38a27ea11d-kube-api-access-pfws6\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.527140 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527050 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-trusted-ca-bundle\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.527140 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527102 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd61feb2-effd-4c91-a9af-2c04eea7a815-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-qzsz6\" (UID: \"cd61feb2-effd-4c91-a9af-2c04eea7a815\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.527140 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527134 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl78p\" (UniqueName: \"kubernetes.io/projected/cd61feb2-effd-4c91-a9af-2c04eea7a815-kube-api-access-cl78p\") pod \"monitoring-plugin-7b7b7bc8bf-qzsz6\" (UID: \"cd61feb2-effd-4c91-a9af-2c04eea7a815\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.527399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-serving-cert\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.527399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527220 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-oauth-serving-cert\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.527399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527277 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-oauth-config\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.527399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-config\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.527399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527353 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfws6\" (UniqueName: \"kubernetes.io/projected/4e167460-3859-4a32-b1e8-8a38a27ea11d-kube-api-access-pfws6\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.527606 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.527566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-service-ca\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.528813 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.528143 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-trusted-ca-bundle\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.528813 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.528153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-service-ca\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.528813 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.528727 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-oauth-serving-cert\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.528813 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.528777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-config\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.530900 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.530872 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-oauth-config\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.531162 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.531123 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd61feb2-effd-4c91-a9af-2c04eea7a815-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-qzsz6\" (UID: \"cd61feb2-effd-4c91-a9af-2c04eea7a815\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.531361 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.531342 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-serving-cert\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.535620 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.535594 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl78p\" (UniqueName: \"kubernetes.io/projected/cd61feb2-effd-4c91-a9af-2c04eea7a815-kube-api-access-cl78p\") pod \"monitoring-plugin-7b7b7bc8bf-qzsz6\" (UID: \"cd61feb2-effd-4c91-a9af-2c04eea7a815\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.536322 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.536295 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfws6\" (UniqueName: \"kubernetes.io/projected/4e167460-3859-4a32-b1e8-8a38a27ea11d-kube-api-access-pfws6\") pod \"console-55b6fc788c-gvj9k\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.638209 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.638170 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:51.652039 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.652011 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:28:51.655332 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.655244 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" event={"ID":"bcf151ce-e42d-47f3-a732-d982b179c7e7","Type":"ContainerStarted","Data":"7e06b462e038d2215c5dd1764b6a63d55f1a1a5df589b9c0b8c735b77b231793"} Jul 17 21:28:51.658463 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.657870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g4x5v" event={"ID":"29753f77-c24d-42c7-8518-24fd9698d3ac","Type":"ContainerStarted","Data":"93d73945e46a59b7c4ea51cd3ecad0f921cc483fde446f6e303c5990605b2ea0"} Jul 17 21:28:51.658463 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.657924 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g4x5v" event={"ID":"29753f77-c24d-42c7-8518-24fd9698d3ac","Type":"ContainerStarted","Data":"4ad8c607662cdb698fb2680958e5a76e02b7053efbaf06086ea520581f82b251"} Jul 17 21:28:51.675253 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.674717 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g4x5v" podStartSLOduration=67.411047622 podStartE2EDuration="1m8.674686428s" totalImagesPullingTime="1.263638806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:51.674055739 +0000 UTC m=+68.944053461" watchObservedRunningTime="2026-07-17 21:28:51.674686428 +0000 UTC m=+68.944684142" Jul 17 21:28:51.857355 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.857034 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6"] Jul 17 21:28:51.893644 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:51.892275 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b6fc788c-gvj9k"] Jul 17 21:28:52.667263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.666742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6fc788c-gvj9k" event={"ID":"4e167460-3859-4a32-b1e8-8a38a27ea11d","Type":"ContainerStarted","Data":"0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99"} Jul 17 21:28:52.667263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.666800 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6fc788c-gvj9k" event={"ID":"4e167460-3859-4a32-b1e8-8a38a27ea11d","Type":"ContainerStarted","Data":"adefceddadc8b90bb87ce37ab4e4d78e9ae7d7705e3658da735b94f3a3c19a7e"} Jul 17 21:28:52.668118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.668083 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" event={"ID":"cd61feb2-effd-4c91-a9af-2c04eea7a815","Type":"ContainerStarted","Data":"1d52d94b855ac55d8d77e7fdc3ac8e304f443d02e825fd3c5bafd706e20031f0"} Jul 17 21:28:52.692621 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.692562 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55b6fc788c-gvj9k" podStartSLOduration=1.69254248 podStartE2EDuration="1.69254248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:52.688127436 +0000 UTC m=+69.958125133" watchObservedRunningTime="2026-07-17 21:28:52.69254248 +0000 UTC m=+69.962540183" Jul 17 21:28:52.742178 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.742139 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:28:52.748678 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.748649 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.753560 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.753537 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 21:28:52.754040 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.753874 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 21:28:52.754663 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.754641 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-alu6e90gubcgi\"" Jul 17 21:28:52.754787 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.754720 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 21:28:52.754847 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.754801 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 21:28:52.754973 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.754956 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 21:28:52.755044 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.754973 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 21:28:52.755044 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.754993 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 21:28:52.755164 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.755148 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 21:28:52.755221 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.754651 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 21:28:52.755481 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.755465 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 21:28:52.755596 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.755569 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 21:28:52.755710 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.755695 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9h2p6\"" Jul 17 21:28:52.755832 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.755806 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 21:28:52.756150 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.756131 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 21:28:52.761351 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.761318 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 21:28:52.767878 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.767644 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:28:52.845585 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845539 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a14f11d-0671-427b-a932-c69238e6f818-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.845783 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.845783 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.845783 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdpn\" (UniqueName: \"kubernetes.io/projected/2a14f11d-0671-427b-a932-c69238e6f818-kube-api-access-vbdpn\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.845783 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845730 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.845783 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845760 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845845 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845904 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845931 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845949 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.845996 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.846027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846472 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.846072 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846472 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.846089 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846472 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.846114 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846472 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.846145 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a14f11d-0671-427b-a932-c69238e6f818-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.846472 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.846178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-config\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.946986 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.946898 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.946986 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.946944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.946986 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.946985 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947012 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947044 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947143 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947170 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947210 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a14f11d-0671-427b-a932-c69238e6f818-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947327 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-config\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947368 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a14f11d-0671-427b-a932-c69238e6f818-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947403 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947426 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947480 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdpn\" (UniqueName: \"kubernetes.io/projected/2a14f11d-0671-427b-a932-c69238e6f818-kube-api-access-vbdpn\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947527 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.947575 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947567 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.948365 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.947602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.948365 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.948020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.948915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.948884 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.949444 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.949129 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.949444 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.949178 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.950083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.950055 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.950654 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.950594 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.951023 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.950997 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.952151 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.951427 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.952347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a14f11d-0671-427b-a932-c69238e6f818-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.952927 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.953209 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.953340 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.953412 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a14f11d-0671-427b-a932-c69238e6f818-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.954250 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.954343 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.954839 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.954898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.954869 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a14f11d-0671-427b-a932-c69238e6f818-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.955441 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.955421 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-config\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.955493 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.955426 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a14f11d-0671-427b-a932-c69238e6f818-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:52.958903 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:52.958882 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdpn\" (UniqueName: \"kubernetes.io/projected/2a14f11d-0671-427b-a932-c69238e6f818-kube-api-access-vbdpn\") pod \"prometheus-k8s-0\" (UID: \"2a14f11d-0671-427b-a932-c69238e6f818\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:53.062981 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:53.062944 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:28:55.781752 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:55.781562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:55.781752 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:55.781678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:55.782934 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:55.782196 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00f31afe-10c6-46c9-bed8-79a188a026da-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:55.787106 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:55.787045 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00f31afe-10c6-46c9-bed8-79a188a026da-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7hk\" (UID: \"00f31afe-10c6-46c9-bed8-79a188a026da\") " pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:55.882256 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:55.882202 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jn2hx\"" Jul 17 21:28:55.894491 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:55.893310 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb7hk" Jul 17 21:28:55.894491 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:55.893774 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 21:28:55.902713 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:55.902677 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a14f11d_0671_427b_a932_c69238e6f818.slice/crio-926c587fcc53ef396584d0edba9b33902a862268343a3c95375facd95ceaf616 WatchSource:0}: Error finding container 926c587fcc53ef396584d0edba9b33902a862268343a3c95375facd95ceaf616: Status 404 returned error can't find the container with id 926c587fcc53ef396584d0edba9b33902a862268343a3c95375facd95ceaf616 Jul 17 21:28:56.068261 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:56.068206 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f31afe_10c6_46c9_bed8_79a188a026da.slice/crio-ab604b48d17a82d2be28c87e838a694c014557ebbf2f30c144dceedad46f4a9c WatchSource:0}: Error finding container ab604b48d17a82d2be28c87e838a694c014557ebbf2f30c144dceedad46f4a9c: Status 404 returned error can't find the container with id ab604b48d17a82d2be28c87e838a694c014557ebbf2f30c144dceedad46f4a9c Jul 17 21:28:56.072163 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.072095 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7hk"] Jul 17 21:28:56.330988 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.330909 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b6fc788c-gvj9k"] Jul 17 21:28:56.366989 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.366954 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75496cd959-x2qrs"] Jul 17 21:28:56.386725 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.386668 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75496cd959-x2qrs"] Jul 17 21:28:56.386884 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.386780 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.487902 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.487869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-trusted-ca-bundle\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.487902 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.487907 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-service-ca\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.488108 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.487926 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-oauth-config\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.488108 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.487941 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mft\" (UniqueName: \"kubernetes.io/projected/6d51d786-8a9a-4409-acf9-b363f761a2d5-kube-api-access-47mft\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.488108 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.488081 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-serving-cert\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.488205 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.488119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-oauth-serving-cert\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.488205 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.488164 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-config\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.589113 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-serving-cert\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.589159 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-oauth-serving-cert\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.589214 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-config\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.589297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-trusted-ca-bundle\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.589327 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-service-ca\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.589353 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-oauth-config\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.589376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mft\" (UniqueName: \"kubernetes.io/projected/6d51d786-8a9a-4409-acf9-b363f761a2d5-kube-api-access-47mft\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.590486 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-config\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.591591 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-service-ca\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.592292 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-trusted-ca-bundle\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.592925 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.592816 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-oauth-serving-cert\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.594327 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.594276 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-serving-cert\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.595989 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.595951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-oauth-config\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.601440 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.601394 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mft\" (UniqueName: \"kubernetes.io/projected/6d51d786-8a9a-4409-acf9-b363f761a2d5-kube-api-access-47mft\") pod \"console-75496cd959-x2qrs\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.689514 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.689472 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-467kv" event={"ID":"b3219454-f5d7-41d3-8ad9-662ae16c50b2","Type":"ContainerStarted","Data":"bc3e8f97bb81b975ef04725577667c5975277d86947092879d36352eba5afa8f"} Jul 17 21:28:56.689804 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.689785 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:28:56.691624 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.691596 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" event={"ID":"cd61feb2-effd-4c91-a9af-2c04eea7a815","Type":"ContainerStarted","Data":"87faf33e9d15aedff66c806efa849ea43a6d8063cdb6aec08a3a43dcd5a44265"} Jul 17 21:28:56.693799 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.693776 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:56.696661 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.696632 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7hk" event={"ID":"00f31afe-10c6-46c9-bed8-79a188a026da","Type":"ContainerStarted","Data":"ab604b48d17a82d2be28c87e838a694c014557ebbf2f30c144dceedad46f4a9c"} Jul 17 21:28:56.699254 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.698587 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbgxj" event={"ID":"4f6911d6-a498-4c63-bb37-156b93e3a189","Type":"ContainerStarted","Data":"cfebdc021f5df89651987c56a5661a1c81383bcfe0bd6bb46eca5db3bc9dd7cf"} Jul 17 21:28:56.699254 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.699059 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:28:56.700544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.700364 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" Jul 17 21:28:56.702950 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.702339 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" event={"ID":"bcf151ce-e42d-47f3-a732-d982b179c7e7","Type":"ContainerStarted","Data":"65349420ac3b1c20444de7208e2dfa611d3e5ec826c229a8ec34b9d12e05f611"} Jul 17 21:28:56.705045 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.704951 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerStarted","Data":"926c587fcc53ef396584d0edba9b33902a862268343a3c95375facd95ceaf616"} Jul 17 21:28:56.711641 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.711266 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-467kv" podStartSLOduration=67.408663455 podStartE2EDuration="1m13.7112489s" totalImagesPullingTime="6.302585445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:56.710377181 +0000 UTC m=+73.980374908" watchObservedRunningTime="2026-07-17 21:28:56.7112489 +0000 UTC m=+73.981246596" Jul 17 21:28:56.727554 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.727505 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xbgxj" podStartSLOduration=67.386459286 podStartE2EDuration="1m13.727484601s" totalImagesPullingTime="6.341025315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:56.725848973 +0000 UTC m=+73.995846677" watchObservedRunningTime="2026-07-17 21:28:56.727484601 +0000 UTC m=+73.997482286" Jul 17 21:28:56.763851 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:56.763530 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" podStartSLOduration=2.354784417 podStartE2EDuration="6.763507345s" totalImagesPullingTime="4.408722928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:56.745086144 +0000 UTC m=+74.015083848" watchObservedRunningTime="2026-07-17 21:28:56.763507345 +0000 UTC m=+74.033505048" Jul 17 21:28:57.301484 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:57.301428 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qzsz6" podStartSLOduration=2.43465954 podStartE2EDuration="6.301408828s" totalImagesPullingTime="3.866749288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:56.765512826 +0000 UTC m=+74.035510531" watchObservedRunningTime="2026-07-17 21:28:57.301408828 +0000 UTC m=+74.571406533" Jul 17 21:28:57.302039 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:57.302008 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75496cd959-x2qrs"] Jul 17 21:28:57.305017 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:28:57.304987 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d51d786_8a9a_4409_acf9_b363f761a2d5.slice/crio-d4acd9092fdc2155f829c6ae67539646ce7053914c932e972fdd3ff56a000b84 WatchSource:0}: Error finding container d4acd9092fdc2155f829c6ae67539646ce7053914c932e972fdd3ff56a000b84: Status 404 returned error can't find the container with id d4acd9092fdc2155f829c6ae67539646ce7053914c932e972fdd3ff56a000b84 Jul 17 21:28:57.712442 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:57.712407 2566 generic.go:356] "Generic (PLEG): container finished" podID="2a14f11d-0671-427b-a932-c69238e6f818" containerID="7d226ea1ee0d7b5e6cce89bb71994d91cc831d28065234eb3acd1adaaa183a0a" exitCode=0 Jul 17 21:28:57.712590 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:57.712534 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerDied","Data":"7d226ea1ee0d7b5e6cce89bb71994d91cc831d28065234eb3acd1adaaa183a0a"} Jul 17 21:28:57.715056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:57.714987 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75496cd959-x2qrs" event={"ID":"6d51d786-8a9a-4409-acf9-b363f761a2d5","Type":"ContainerStarted","Data":"a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30"} Jul 17 21:28:57.715056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:57.715032 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75496cd959-x2qrs" event={"ID":"6d51d786-8a9a-4409-acf9-b363f761a2d5","Type":"ContainerStarted","Data":"d4acd9092fdc2155f829c6ae67539646ce7053914c932e972fdd3ff56a000b84"} Jul 17 21:28:57.772493 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:28:57.772443 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75496cd959-x2qrs" podStartSLOduration=1.7724272239999999 podStartE2EDuration="1.772427224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:57.770993682 +0000 UTC m=+75.040991388" watchObservedRunningTime="2026-07-17 21:28:57.772427224 +0000 UTC m=+75.042424927" Jul 17 21:29:01.653248 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:01.653195 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:29:03.742953 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:03.742509 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7hk" event={"ID":"00f31afe-10c6-46c9-bed8-79a188a026da","Type":"ContainerStarted","Data":"8504ebd2a495e25ce7f042211c32f953e1c12ed8e71a33faf5bea9c745185335"} Jul 17 21:29:03.745698 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:03.745674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerStarted","Data":"6a8ce9606664a977e0f0b67c83af8ae08618c88f58da9a81149d3a5679f7a9a3"} Jul 17 21:29:03.745809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:03.745748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerStarted","Data":"1579006b616eb44bb3b7a82ecc933e72ad3791562dbd173df2bca61027bc3766"} Jul 17 21:29:06.700094 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:06.700063 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:29:06.700465 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:06.700143 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:29:06.704817 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:06.704796 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:29:06.761637 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:06.761574 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:29:06.817739 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:06.817704 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fc5687d-vjvjr"] Jul 17 21:29:07.766560 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:07.766526 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerStarted","Data":"b5fb2953b301b812795ed2b77370aee7a7be63d8c1e955b6a9eae3c05c2e5fc4"} Jul 17 21:29:07.767052 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:07.766605 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerStarted","Data":"c3ec7bfd011c4e9edbfee3134f94470981ee55eb7440fc90cdbbd3ab1e8e1d8d"} Jul 17 21:29:07.767052 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:07.766646 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerStarted","Data":"405159f4f852cfc95394d70bfd7aa253c52de311e40ea492a6627c593d61d5ab"} Jul 17 21:29:07.767052 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:07.766686 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a14f11d-0671-427b-a932-c69238e6f818","Type":"ContainerStarted","Data":"501ff3545e0d0ffc506840486f4f04134517815b21a06194dd5a7b3dfdc3aece"} Jul 17 21:29:07.768540 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:07.768507 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7hk" event={"ID":"00f31afe-10c6-46c9-bed8-79a188a026da","Type":"ContainerStarted","Data":"36a055c1cecce9df1c1388f867623ccdfc7e61abec69d45b8ada078ee800cf50"} Jul 17 21:29:07.797561 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:07.797495 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.666029493 podStartE2EDuration="15.79747402s" totalImagesPullingTime="10.101234565s" totalInitContainerRuntime="30.209962ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:29:07.793394911 +0000 UTC m=+85.063392664" watchObservedRunningTime="2026-07-17 21:29:07.79747402 +0000 UTC m=+85.067471726" Jul 17 21:29:08.063593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:08.063504 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:29:11.121886 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:11.121803 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:29:11.121886 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:11.121872 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:29:11.789471 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:11.789419 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7hk" event={"ID":"00f31afe-10c6-46c9-bed8-79a188a026da","Type":"ContainerStarted","Data":"11e9f7c74aee46b9ed7f5be36f2c9b83f72656fb902ab1254e8ded5722f59dea"} Jul 17 21:29:11.810158 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:11.810102 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mb7hk" podStartSLOduration=18.220221361 podStartE2EDuration="32.810087275s" totalImagesPullingTime="14.589865914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:28:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:29:11.808357645 +0000 UTC m=+89.078355374" watchObservedRunningTime="2026-07-17 21:29:11.810087275 +0000 UTC m=+89.080084981" Jul 17 21:29:21.354550 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.354481 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55b6fc788c-gvj9k" podUID="4e167460-3859-4a32-b1e8-8a38a27ea11d" containerName="console" containerID="cri-o://0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99" gracePeriod=15 Jul 17 21:29:21.598698 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.598676 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:29:21.619720 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.619638 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfws6\" (UniqueName: \"kubernetes.io/projected/4e167460-3859-4a32-b1e8-8a38a27ea11d-kube-api-access-pfws6\") pod \"4e167460-3859-4a32-b1e8-8a38a27ea11d\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " Jul 17 21:29:21.619720 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.619675 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-oauth-config\") pod \"4e167460-3859-4a32-b1e8-8a38a27ea11d\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " Jul 17 21:29:21.619720 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.619694 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-config\") pod \"4e167460-3859-4a32-b1e8-8a38a27ea11d\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " Jul 17 21:29:21.619983 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.619746 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-service-ca\") pod \"4e167460-3859-4a32-b1e8-8a38a27ea11d\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " Jul 17 21:29:21.619983 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.619808 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-oauth-serving-cert\") pod \"4e167460-3859-4a32-b1e8-8a38a27ea11d\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " Jul 17 21:29:21.619983 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.619837 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-trusted-ca-bundle\") pod \"4e167460-3859-4a32-b1e8-8a38a27ea11d\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " Jul 17 21:29:21.620370 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620333 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-config" (OuterVolumeSpecName: "console-config") pod "4e167460-3859-4a32-b1e8-8a38a27ea11d" (UID: "4e167460-3859-4a32-b1e8-8a38a27ea11d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:21.620489 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620372 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e167460-3859-4a32-b1e8-8a38a27ea11d" (UID: "4e167460-3859-4a32-b1e8-8a38a27ea11d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:21.620489 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620408 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-serving-cert\") pod \"4e167460-3859-4a32-b1e8-8a38a27ea11d\" (UID: \"4e167460-3859-4a32-b1e8-8a38a27ea11d\") " Jul 17 21:29:21.620599 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620490 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e167460-3859-4a32-b1e8-8a38a27ea11d" (UID: "4e167460-3859-4a32-b1e8-8a38a27ea11d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:21.620900 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620877 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:21.621002 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620907 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-oauth-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:21.621002 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620922 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-trusted-ca-bundle\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:21.621102 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.620996 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e167460-3859-4a32-b1e8-8a38a27ea11d" (UID: "4e167460-3859-4a32-b1e8-8a38a27ea11d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:21.622830 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.622783 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e167460-3859-4a32-b1e8-8a38a27ea11d" (UID: "4e167460-3859-4a32-b1e8-8a38a27ea11d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:29:21.623079 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.623054 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e167460-3859-4a32-b1e8-8a38a27ea11d-kube-api-access-pfws6" (OuterVolumeSpecName: "kube-api-access-pfws6") pod "4e167460-3859-4a32-b1e8-8a38a27ea11d" (UID: "4e167460-3859-4a32-b1e8-8a38a27ea11d"). InnerVolumeSpecName "kube-api-access-pfws6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:29:21.623141 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.623061 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e167460-3859-4a32-b1e8-8a38a27ea11d" (UID: "4e167460-3859-4a32-b1e8-8a38a27ea11d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:29:21.722299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.722266 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfws6\" (UniqueName: \"kubernetes.io/projected/4e167460-3859-4a32-b1e8-8a38a27ea11d-kube-api-access-pfws6\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:21.722299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.722295 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-oauth-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:21.722299 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.722306 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e167460-3859-4a32-b1e8-8a38a27ea11d-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:21.722517 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.722314 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e167460-3859-4a32-b1e8-8a38a27ea11d-console-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:21.830965 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.830926 2566 generic.go:356] "Generic (PLEG): container finished" podID="4e167460-3859-4a32-b1e8-8a38a27ea11d" containerID="0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99" exitCode=0 Jul 17 21:29:21.831118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.830975 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b6fc788c-gvj9k" Jul 17 21:29:21.831118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.831012 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6fc788c-gvj9k" event={"ID":"4e167460-3859-4a32-b1e8-8a38a27ea11d","Type":"ContainerDied","Data":"0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99"} Jul 17 21:29:21.831118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.831059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b6fc788c-gvj9k" event={"ID":"4e167460-3859-4a32-b1e8-8a38a27ea11d","Type":"ContainerDied","Data":"adefceddadc8b90bb87ce37ab4e4d78e9ae7d7705e3658da735b94f3a3c19a7e"} Jul 17 21:29:21.831118 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.831079 2566 scope.go:117] "RemoveContainer" containerID="0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99" Jul 17 21:29:21.841628 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.841605 2566 scope.go:117] "RemoveContainer" containerID="0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99" Jul 17 21:29:21.841916 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:29:21.841888 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99\": container with ID starting with 0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99 not found: ID does not exist" containerID="0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99" Jul 17 21:29:21.841979 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.841920 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99"} err="failed to get container status \"0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99\": rpc error: code = NotFound desc = could not find container \"0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99\": container with ID starting with 0c3c071ee18830c35598116d7ffc8e3e6cb22fe71f125848f06b21d6d7745c99 not found: ID does not exist" Jul 17 21:29:21.855370 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.855343 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b6fc788c-gvj9k"] Jul 17 21:29:21.860531 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:21.860509 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55b6fc788c-gvj9k"] Jul 17 21:29:23.297650 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:23.297618 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e167460-3859-4a32-b1e8-8a38a27ea11d" path="/var/lib/kubelet/pods/4e167460-3859-4a32-b1e8-8a38a27ea11d/volumes" Jul 17 21:29:27.717298 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:27.717270 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-467kv" Jul 17 21:29:31.126697 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:31.126670 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:29:31.130612 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:31.130591 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-589fb9d4f5-hcd2p" Jul 17 21:29:31.854415 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:31.854341 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54fc5687d-vjvjr" podUID="044d02a6-5527-4cb2-bcfa-90632edbb4a5" containerName="console" containerID="cri-o://3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276" gracePeriod=15 Jul 17 21:29:32.157050 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.157020 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:29:32.214183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.213478 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-oauth-config\") pod \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " Jul 17 21:29:32.214183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.213530 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-trusted-ca-bundle\") pod \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " Jul 17 21:29:32.214183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.213603 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-oauth-serving-cert\") pod \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " Jul 17 21:29:32.214183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.213640 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-serving-cert\") pod \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " Jul 17 21:29:32.214183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.213675 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-config\") pod \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " Jul 17 21:29:32.214183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.213712 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpltz\" (UniqueName: \"kubernetes.io/projected/044d02a6-5527-4cb2-bcfa-90632edbb4a5-kube-api-access-tpltz\") pod \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " Jul 17 21:29:32.214183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.213745 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-service-ca\") pod \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\" (UID: \"044d02a6-5527-4cb2-bcfa-90632edbb4a5\") " Jul 17 21:29:32.215111 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.214783 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-service-ca" (OuterVolumeSpecName: "service-ca") pod "044d02a6-5527-4cb2-bcfa-90632edbb4a5" (UID: "044d02a6-5527-4cb2-bcfa-90632edbb4a5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:32.218884 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.218618 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-config" (OuterVolumeSpecName: "console-config") pod "044d02a6-5527-4cb2-bcfa-90632edbb4a5" (UID: "044d02a6-5527-4cb2-bcfa-90632edbb4a5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:32.221294 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.221015 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "044d02a6-5527-4cb2-bcfa-90632edbb4a5" (UID: "044d02a6-5527-4cb2-bcfa-90632edbb4a5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:32.221294 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.221257 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "044d02a6-5527-4cb2-bcfa-90632edbb4a5" (UID: "044d02a6-5527-4cb2-bcfa-90632edbb4a5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:29:32.229491 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.228665 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "044d02a6-5527-4cb2-bcfa-90632edbb4a5" (UID: "044d02a6-5527-4cb2-bcfa-90632edbb4a5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:29:32.230123 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.230083 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044d02a6-5527-4cb2-bcfa-90632edbb4a5-kube-api-access-tpltz" (OuterVolumeSpecName: "kube-api-access-tpltz") pod "044d02a6-5527-4cb2-bcfa-90632edbb4a5" (UID: "044d02a6-5527-4cb2-bcfa-90632edbb4a5"). InnerVolumeSpecName "kube-api-access-tpltz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:29:32.231483 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.231446 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "044d02a6-5527-4cb2-bcfa-90632edbb4a5" (UID: "044d02a6-5527-4cb2-bcfa-90632edbb4a5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:29:32.315303 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.315269 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:32.315303 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.315300 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpltz\" (UniqueName: \"kubernetes.io/projected/044d02a6-5527-4cb2-bcfa-90632edbb4a5-kube-api-access-tpltz\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:32.315303 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.315310 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:32.315519 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.315318 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-oauth-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:32.315519 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.315327 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-trusted-ca-bundle\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:32.315519 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.315334 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/044d02a6-5527-4cb2-bcfa-90632edbb4a5-oauth-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:32.315519 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.315342 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/044d02a6-5527-4cb2-bcfa-90632edbb4a5-console-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:29:32.874640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.874603 2566 generic.go:356] "Generic (PLEG): container finished" podID="044d02a6-5527-4cb2-bcfa-90632edbb4a5" containerID="3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276" exitCode=0 Jul 17 21:29:32.874809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.874653 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fc5687d-vjvjr" Jul 17 21:29:32.874809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.874663 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fc5687d-vjvjr" event={"ID":"044d02a6-5527-4cb2-bcfa-90632edbb4a5","Type":"ContainerDied","Data":"3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276"} Jul 17 21:29:32.874809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.874711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fc5687d-vjvjr" event={"ID":"044d02a6-5527-4cb2-bcfa-90632edbb4a5","Type":"ContainerDied","Data":"614e0d6256d4e3c54c2b80cf33e11b42324df17d2bed89b0061722d579e5a634"} Jul 17 21:29:32.874809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.874740 2566 scope.go:117] "RemoveContainer" containerID="3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276" Jul 17 21:29:32.883258 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.883219 2566 scope.go:117] "RemoveContainer" containerID="3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276" Jul 17 21:29:32.883521 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:29:32.883503 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276\": container with ID starting with 3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276 not found: ID does not exist" containerID="3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276" Jul 17 21:29:32.883573 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.883530 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276"} err="failed to get container status \"3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276\": rpc error: code = NotFound desc = could not find container \"3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276\": container with ID starting with 3ef497a3c0ac5a6fedfe0bf22e478aae00c99f1cb75fbfd5a9ef7ca5f6ff0276 not found: ID does not exist" Jul 17 21:29:32.896072 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.896046 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54fc5687d-vjvjr"] Jul 17 21:29:32.899752 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:32.899732 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54fc5687d-vjvjr"] Jul 17 21:29:33.297146 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:33.297111 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044d02a6-5527-4cb2-bcfa-90632edbb4a5" path="/var/lib/kubelet/pods/044d02a6-5527-4cb2-bcfa-90632edbb4a5/volumes" Jul 17 21:29:53.063474 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:53.063416 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:29:53.082810 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:53.082779 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:29:53.967001 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:29:53.966971 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 21:30:06.066010 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.065971 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d4ffc6fb-gvgsp"] Jul 17 21:30:06.066390 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.066333 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044d02a6-5527-4cb2-bcfa-90632edbb4a5" containerName="console" Jul 17 21:30:06.066390 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.066348 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="044d02a6-5527-4cb2-bcfa-90632edbb4a5" containerName="console" Jul 17 21:30:06.066390 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.066360 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e167460-3859-4a32-b1e8-8a38a27ea11d" containerName="console" Jul 17 21:30:06.066390 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.066366 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e167460-3859-4a32-b1e8-8a38a27ea11d" containerName="console" Jul 17 21:30:06.066529 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.066432 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="044d02a6-5527-4cb2-bcfa-90632edbb4a5" containerName="console" Jul 17 21:30:06.066529 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.066442 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e167460-3859-4a32-b1e8-8a38a27ea11d" containerName="console" Jul 17 21:30:06.069621 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.069602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.084944 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.084919 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d4ffc6fb-gvgsp"] Jul 17 21:30:06.095935 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.095901 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-oauth-serving-cert\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.096110 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.095952 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-oauth-config\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.096110 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.096016 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-config\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.096110 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.096037 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-service-ca\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.096110 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.096065 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-serving-cert\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.096310 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.096185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-trusted-ca-bundle\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.096310 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.096248 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wf86\" (UniqueName: \"kubernetes.io/projected/219b3cd0-0636-4ecd-85a7-06976b5b2900-kube-api-access-5wf86\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.196824 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.196784 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-trusted-ca-bundle\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.196824 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.196827 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wf86\" (UniqueName: \"kubernetes.io/projected/219b3cd0-0636-4ecd-85a7-06976b5b2900-kube-api-access-5wf86\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197059 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.196858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-oauth-serving-cert\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197059 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.196900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-oauth-config\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197059 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.196964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-config\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197059 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.196985 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-service-ca\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197059 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.197020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-serving-cert\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197903 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.197806 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-oauth-serving-cert\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197903 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.197826 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-trusted-ca-bundle\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.197903 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.197859 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-service-ca\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.198122 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.198012 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-config\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.199671 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.199652 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-serving-cert\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.199767 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.199675 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-oauth-config\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.208481 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.208459 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wf86\" (UniqueName: \"kubernetes.io/projected/219b3cd0-0636-4ecd-85a7-06976b5b2900-kube-api-access-5wf86\") pod \"console-57d4ffc6fb-gvgsp\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.379476 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.379384 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:06.514548 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.514512 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d4ffc6fb-gvgsp"] Jul 17 21:30:06.518817 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:30:06.518786 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219b3cd0_0636_4ecd_85a7_06976b5b2900.slice/crio-b283e4cc035aec591a5060ec661239b3c08242772caae58b1e868807d86e9629 WatchSource:0}: Error finding container b283e4cc035aec591a5060ec661239b3c08242772caae58b1e868807d86e9629: Status 404 returned error can't find the container with id b283e4cc035aec591a5060ec661239b3c08242772caae58b1e868807d86e9629 Jul 17 21:30:06.997057 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.997019 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d4ffc6fb-gvgsp" event={"ID":"219b3cd0-0636-4ecd-85a7-06976b5b2900","Type":"ContainerStarted","Data":"c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024"} Jul 17 21:30:06.997297 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:06.997081 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d4ffc6fb-gvgsp" event={"ID":"219b3cd0-0636-4ecd-85a7-06976b5b2900","Type":"ContainerStarted","Data":"b283e4cc035aec591a5060ec661239b3c08242772caae58b1e868807d86e9629"} Jul 17 21:30:07.016871 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:07.016812 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57d4ffc6fb-gvgsp" podStartSLOduration=1.016794859 podStartE2EDuration="1.016794859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:30:07.015439331 +0000 UTC m=+144.285437036" watchObservedRunningTime="2026-07-17 21:30:07.016794859 +0000 UTC m=+144.286792563" Jul 17 21:30:16.380197 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:16.380163 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:16.380754 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:16.380466 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:16.385252 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:16.385213 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:17.037317 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:17.037292 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:30:17.121273 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:17.121244 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75496cd959-x2qrs"] Jul 17 21:30:42.145074 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.145036 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75496cd959-x2qrs" podUID="6d51d786-8a9a-4409-acf9-b363f761a2d5" containerName="console" containerID="cri-o://a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30" gracePeriod=15 Jul 17 21:30:42.389333 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.389310 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:30:42.422456 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422428 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47mft\" (UniqueName: \"kubernetes.io/projected/6d51d786-8a9a-4409-acf9-b363f761a2d5-kube-api-access-47mft\") pod \"6d51d786-8a9a-4409-acf9-b363f761a2d5\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " Jul 17 21:30:42.422639 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422465 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-config\") pod \"6d51d786-8a9a-4409-acf9-b363f761a2d5\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " Jul 17 21:30:42.422639 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422497 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-oauth-config\") pod \"6d51d786-8a9a-4409-acf9-b363f761a2d5\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " Jul 17 21:30:42.422639 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422528 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-service-ca\") pod \"6d51d786-8a9a-4409-acf9-b363f761a2d5\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " Jul 17 21:30:42.422639 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422569 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-serving-cert\") pod \"6d51d786-8a9a-4409-acf9-b363f761a2d5\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " Jul 17 21:30:42.422639 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422586 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-trusted-ca-bundle\") pod \"6d51d786-8a9a-4409-acf9-b363f761a2d5\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " Jul 17 21:30:42.422639 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422614 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-oauth-serving-cert\") pod \"6d51d786-8a9a-4409-acf9-b363f761a2d5\" (UID: \"6d51d786-8a9a-4409-acf9-b363f761a2d5\") " Jul 17 21:30:42.422938 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.422888 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-config" (OuterVolumeSpecName: "console-config") pod "6d51d786-8a9a-4409-acf9-b363f761a2d5" (UID: "6d51d786-8a9a-4409-acf9-b363f761a2d5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:30:42.423252 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.423171 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6d51d786-8a9a-4409-acf9-b363f761a2d5" (UID: "6d51d786-8a9a-4409-acf9-b363f761a2d5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:30:42.423521 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.423180 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-service-ca" (OuterVolumeSpecName: "service-ca") pod "6d51d786-8a9a-4409-acf9-b363f761a2d5" (UID: "6d51d786-8a9a-4409-acf9-b363f761a2d5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:30:42.423521 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.423497 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6d51d786-8a9a-4409-acf9-b363f761a2d5" (UID: "6d51d786-8a9a-4409-acf9-b363f761a2d5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:30:42.424881 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.424853 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6d51d786-8a9a-4409-acf9-b363f761a2d5" (UID: "6d51d786-8a9a-4409-acf9-b363f761a2d5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:30:42.424881 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.424869 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6d51d786-8a9a-4409-acf9-b363f761a2d5" (UID: "6d51d786-8a9a-4409-acf9-b363f761a2d5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:30:42.425049 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.424871 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d51d786-8a9a-4409-acf9-b363f761a2d5-kube-api-access-47mft" (OuterVolumeSpecName: "kube-api-access-47mft") pod "6d51d786-8a9a-4409-acf9-b363f761a2d5" (UID: "6d51d786-8a9a-4409-acf9-b363f761a2d5"). InnerVolumeSpecName "kube-api-access-47mft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:30:42.523556 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.523527 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:30:42.523556 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.523553 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:30:42.523556 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.523562 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-trusted-ca-bundle\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:30:42.523765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.523573 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-oauth-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:30:42.523765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.523584 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mft\" (UniqueName: \"kubernetes.io/projected/6d51d786-8a9a-4409-acf9-b363f761a2d5-kube-api-access-47mft\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:30:42.523765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.523592 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:30:42.523765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:42.523600 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6d51d786-8a9a-4409-acf9-b363f761a2d5-console-oauth-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:30:43.126725 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.126693 2566 generic.go:356] "Generic (PLEG): container finished" podID="6d51d786-8a9a-4409-acf9-b363f761a2d5" containerID="a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30" exitCode=0 Jul 17 21:30:43.126927 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.126754 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75496cd959-x2qrs" Jul 17 21:30:43.126927 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.126775 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75496cd959-x2qrs" event={"ID":"6d51d786-8a9a-4409-acf9-b363f761a2d5","Type":"ContainerDied","Data":"a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30"} Jul 17 21:30:43.126927 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.126829 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75496cd959-x2qrs" event={"ID":"6d51d786-8a9a-4409-acf9-b363f761a2d5","Type":"ContainerDied","Data":"d4acd9092fdc2155f829c6ae67539646ce7053914c932e972fdd3ff56a000b84"} Jul 17 21:30:43.126927 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.126849 2566 scope.go:117] "RemoveContainer" containerID="a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30" Jul 17 21:30:43.135700 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.135680 2566 scope.go:117] "RemoveContainer" containerID="a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30" Jul 17 21:30:43.135985 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:30:43.135967 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30\": container with ID starting with a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30 not found: ID does not exist" containerID="a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30" Jul 17 21:30:43.136036 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.135994 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30"} err="failed to get container status \"a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30\": rpc error: code = NotFound desc = could not find container \"a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30\": container with ID starting with a82c8c68fed8dc83c11d58d6d5885ed137a82913612c8495f73ec65bca42fc30 not found: ID does not exist" Jul 17 21:30:43.148516 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.148486 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75496cd959-x2qrs"] Jul 17 21:30:43.152039 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.152016 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75496cd959-x2qrs"] Jul 17 21:30:43.296847 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:30:43.296806 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d51d786-8a9a-4409-acf9-b363f761a2d5" path="/var/lib/kubelet/pods/6d51d786-8a9a-4409-acf9-b363f761a2d5/volumes" Jul 17 21:32:09.480044 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.479968 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jgr7m"] Jul 17 21:32:09.480511 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.480305 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d51d786-8a9a-4409-acf9-b363f761a2d5" containerName="console" Jul 17 21:32:09.480511 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.480318 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d51d786-8a9a-4409-acf9-b363f761a2d5" containerName="console" Jul 17 21:32:09.480511 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.480372 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d51d786-8a9a-4409-acf9-b363f761a2d5" containerName="console" Jul 17 21:32:09.482176 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.482159 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.484963 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.484938 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 21:32:09.485922 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.485906 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vm2s9\"" Jul 17 21:32:09.485996 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.485914 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 21:32:09.493147 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.493124 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jgr7m"] Jul 17 21:32:09.618649 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.618612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2kx\" (UniqueName: \"kubernetes.io/projected/7b775d51-d78a-42f5-9aca-29dbcd497414-kube-api-access-sh2kx\") pod \"cert-manager-55b4d954d5-jgr7m\" (UID: \"7b775d51-d78a-42f5-9aca-29dbcd497414\") " pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.618649 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.618656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b775d51-d78a-42f5-9aca-29dbcd497414-bound-sa-token\") pod \"cert-manager-55b4d954d5-jgr7m\" (UID: \"7b775d51-d78a-42f5-9aca-29dbcd497414\") " pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.720001 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.719963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2kx\" (UniqueName: \"kubernetes.io/projected/7b775d51-d78a-42f5-9aca-29dbcd497414-kube-api-access-sh2kx\") pod \"cert-manager-55b4d954d5-jgr7m\" (UID: \"7b775d51-d78a-42f5-9aca-29dbcd497414\") " pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.720001 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.720005 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b775d51-d78a-42f5-9aca-29dbcd497414-bound-sa-token\") pod \"cert-manager-55b4d954d5-jgr7m\" (UID: \"7b775d51-d78a-42f5-9aca-29dbcd497414\") " pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.729083 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.729052 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b775d51-d78a-42f5-9aca-29dbcd497414-bound-sa-token\") pod \"cert-manager-55b4d954d5-jgr7m\" (UID: \"7b775d51-d78a-42f5-9aca-29dbcd497414\") " pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.729329 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.729311 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2kx\" (UniqueName: \"kubernetes.io/projected/7b775d51-d78a-42f5-9aca-29dbcd497414-kube-api-access-sh2kx\") pod \"cert-manager-55b4d954d5-jgr7m\" (UID: \"7b775d51-d78a-42f5-9aca-29dbcd497414\") " pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.797416 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.797330 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jgr7m" Jul 17 21:32:09.948852 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:09.948821 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jgr7m"] Jul 17 21:32:09.951979 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:32:09.951944 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b775d51_d78a_42f5_9aca_29dbcd497414.slice/crio-4d2f7687363e01f4223f47ef77f73c1c5993e5ccdad7dded9d6fb2548f028fba WatchSource:0}: Error finding container 4d2f7687363e01f4223f47ef77f73c1c5993e5ccdad7dded9d6fb2548f028fba: Status 404 returned error can't find the container with id 4d2f7687363e01f4223f47ef77f73c1c5993e5ccdad7dded9d6fb2548f028fba Jul 17 21:32:10.437905 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:10.437872 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jgr7m" event={"ID":"7b775d51-d78a-42f5-9aca-29dbcd497414","Type":"ContainerStarted","Data":"4d2f7687363e01f4223f47ef77f73c1c5993e5ccdad7dded9d6fb2548f028fba"} Jul 17 21:32:11.989842 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:11.989804 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn"] Jul 17 21:32:11.994882 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:11.994856 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:12.002762 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.002733 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 21:32:12.003854 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.003825 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gz9hz\"" Jul 17 21:32:12.003997 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.003948 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:32:12.006294 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.006270 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn"] Jul 17 21:32:12.143042 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.143004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzxf\" (UniqueName: \"kubernetes.io/projected/4d6b63ad-6de7-4e09-8634-6d9d2bd0f289-kube-api-access-wdzxf\") pod \"openshift-lws-operator-fd8ccff4c-ck9vn\" (UID: \"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:12.143212 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.143069 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d6b63ad-6de7-4e09-8634-6d9d2bd0f289-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ck9vn\" (UID: \"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:12.244014 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.243927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d6b63ad-6de7-4e09-8634-6d9d2bd0f289-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ck9vn\" (UID: \"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:12.244207 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.244023 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzxf\" (UniqueName: \"kubernetes.io/projected/4d6b63ad-6de7-4e09-8634-6d9d2bd0f289-kube-api-access-wdzxf\") pod \"openshift-lws-operator-fd8ccff4c-ck9vn\" (UID: \"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:12.244458 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.244424 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d6b63ad-6de7-4e09-8634-6d9d2bd0f289-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ck9vn\" (UID: \"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:12.254245 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.254195 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzxf\" (UniqueName: \"kubernetes.io/projected/4d6b63ad-6de7-4e09-8634-6d9d2bd0f289-kube-api-access-wdzxf\") pod \"openshift-lws-operator-fd8ccff4c-ck9vn\" (UID: \"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:12.310183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:12.310149 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" Jul 17 21:32:13.134558 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:13.134535 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn"] Jul 17 21:32:13.137557 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:32:13.137514 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d6b63ad_6de7_4e09_8634_6d9d2bd0f289.slice/crio-db18b3d118a17f2734169dd7a22d5609cc5af7db8e03462b9cd81e1f5836ae16 WatchSource:0}: Error finding container db18b3d118a17f2734169dd7a22d5609cc5af7db8e03462b9cd81e1f5836ae16: Status 404 returned error can't find the container with id db18b3d118a17f2734169dd7a22d5609cc5af7db8e03462b9cd81e1f5836ae16 Jul 17 21:32:13.451696 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:13.451663 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jgr7m" event={"ID":"7b775d51-d78a-42f5-9aca-29dbcd497414","Type":"ContainerStarted","Data":"a7825ea7b0ed3cc4f00b5553a3867e30886f98567710323702b2fd6857dc8e2f"} Jul 17 21:32:13.452991 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:13.452958 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" event={"ID":"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289","Type":"ContainerStarted","Data":"db18b3d118a17f2734169dd7a22d5609cc5af7db8e03462b9cd81e1f5836ae16"} Jul 17 21:32:13.469139 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:13.469083 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jgr7m" podStartSLOduration=1.383690252 podStartE2EDuration="4.46906679s" totalImagesPullingTime="3.085376538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:13.468307166 +0000 UTC m=+270.738304872" watchObservedRunningTime="2026-07-17 21:32:13.46906679 +0000 UTC m=+270.739064494" Jul 17 21:32:16.466324 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:16.466289 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" event={"ID":"4d6b63ad-6de7-4e09-8634-6d9d2bd0f289","Type":"ContainerStarted","Data":"6c17a8ba734d6cb45f9e21fb7c54f14af8f7e5f4929013e9a4725f3009cbad94"} Jul 17 21:32:16.484312 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:16.484260 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ck9vn" podStartSLOduration=3.157751575 podStartE2EDuration="5.48422345s" totalImagesPullingTime="2.326471875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:16.483415584 +0000 UTC m=+273.753413289" watchObservedRunningTime="2026-07-17 21:32:16.48422345 +0000 UTC m=+273.754221153" Jul 17 21:32:34.539504 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.539469 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56"] Jul 17 21:32:34.541932 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.541912 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.544418 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.544391 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 21:32:34.544743 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.544729 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9ws4k\"" Jul 17 21:32:34.544847 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.544827 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 21:32:34.544904 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.544825 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 21:32:34.545136 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.545121 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 21:32:34.584799 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.584768 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56"] Jul 17 21:32:34.633896 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.633861 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.634070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.633916 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.634070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.633955 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6qm\" (UniqueName: \"kubernetes.io/projected/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-kube-api-access-zd6qm\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.634070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.633995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.634070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.634024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.734388 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.734358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6qm\" (UniqueName: \"kubernetes.io/projected/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-kube-api-access-zd6qm\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.734388 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.734391 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.734566 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.734424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.734566 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.734454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.734566 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.734499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.734911 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.734881 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-manifests\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.736819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.736798 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-tmp\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.736977 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.736964 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.737042 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.737000 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.748516 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.748488 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6qm\" (UniqueName: \"kubernetes.io/projected/48cf6d79-d7e0-451b-a3c6-53f54049e3b3-kube-api-access-zd6qm\") pod \"opendatahub-operator-controller-manager-964b768d-v2w56\" (UID: \"48cf6d79-d7e0-451b-a3c6-53f54049e3b3\") " pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.852547 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.852450 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:34.988770 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:34.988684 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56"] Jul 17 21:32:34.991650 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:32:34.991616 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cf6d79_d7e0_451b_a3c6_53f54049e3b3.slice/crio-f961b8700d97dfe11bfa2272ae70e8bee81e89e805f093586f6d0d1c272e9e4c WatchSource:0}: Error finding container f961b8700d97dfe11bfa2272ae70e8bee81e89e805f093586f6d0d1c272e9e4c: Status 404 returned error can't find the container with id f961b8700d97dfe11bfa2272ae70e8bee81e89e805f093586f6d0d1c272e9e4c Jul 17 21:32:35.534651 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:35.534615 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" event={"ID":"48cf6d79-d7e0-451b-a3c6-53f54049e3b3","Type":"ContainerStarted","Data":"f961b8700d97dfe11bfa2272ae70e8bee81e89e805f093586f6d0d1c272e9e4c"} Jul 17 21:32:38.550449 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:38.550412 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" event={"ID":"48cf6d79-d7e0-451b-a3c6-53f54049e3b3","Type":"ContainerStarted","Data":"db56b2ba2eb4694dfb5482c6cf8f7e0586619cda424a22120ccf74ecfb7062be"} Jul 17 21:32:39.555456 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:39.555417 2566 generic.go:356] "Generic (PLEG): container finished" podID="48cf6d79-d7e0-451b-a3c6-53f54049e3b3" containerID="db56b2ba2eb4694dfb5482c6cf8f7e0586619cda424a22120ccf74ecfb7062be" exitCode=0 Jul 17 21:32:39.555843 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:39.555475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" event={"ID":"48cf6d79-d7e0-451b-a3c6-53f54049e3b3","Type":"ContainerDied","Data":"db56b2ba2eb4694dfb5482c6cf8f7e0586619cda424a22120ccf74ecfb7062be"} Jul 17 21:32:40.561447 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:40.561415 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" event={"ID":"48cf6d79-d7e0-451b-a3c6-53f54049e3b3","Type":"ContainerStarted","Data":"866f0deaf6d9292f45cf735deb95f8e24ee4d20283ea7d61d47627738ade47fb"} Jul 17 21:32:40.561903 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:40.561486 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:40.584894 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:40.584839 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" podStartSLOduration=2.190245512 podStartE2EDuration="6.58482065s" totalImagesPullingTime="3.384462841s" totalInitContainerRuntime="1.010112297s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:40.582461262 +0000 UTC m=+297.852458966" watchObservedRunningTime="2026-07-17 21:32:40.58482065 +0000 UTC m=+297.854818353" Jul 17 21:32:43.105981 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:43.105954 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 21:32:51.568020 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:51.567986 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-964b768d-v2w56" Jul 17 21:32:54.215856 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.215822 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr"] Jul 17 21:32:54.223688 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.223667 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.227411 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.227389 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 21:32:54.227530 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.227411 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 21:32:54.227530 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.227411 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 21:32:54.227530 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.227428 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8kwkm\"" Jul 17 21:32:54.227530 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.227433 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 21:32:54.234400 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.234380 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr"] Jul 17 21:32:54.303364 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.303327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzmc\" (UniqueName: \"kubernetes.io/projected/0d903116-17f2-4664-8e0e-6499258fab01-kube-api-access-4rzmc\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.303532 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.303369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d903116-17f2-4664-8e0e-6499258fab01-tls-certs\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.303532 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.303490 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d903116-17f2-4664-8e0e-6499258fab01-tmp\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.404655 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.404615 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d903116-17f2-4664-8e0e-6499258fab01-tmp\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.404821 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.404679 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzmc\" (UniqueName: \"kubernetes.io/projected/0d903116-17f2-4664-8e0e-6499258fab01-kube-api-access-4rzmc\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.404821 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.404715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d903116-17f2-4664-8e0e-6499258fab01-tls-certs\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.407131 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.407105 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d903116-17f2-4664-8e0e-6499258fab01-tmp\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.407398 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.407380 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d903116-17f2-4664-8e0e-6499258fab01-tls-certs\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.421907 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.421877 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzmc\" (UniqueName: \"kubernetes.io/projected/0d903116-17f2-4664-8e0e-6499258fab01-kube-api-access-4rzmc\") pod \"kube-auth-proxy-788f6fc9d-5mrmr\" (UID: \"0d903116-17f2-4664-8e0e-6499258fab01\") " pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.533530 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.533442 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" Jul 17 21:32:54.669835 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.669804 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr"] Jul 17 21:32:54.673041 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:32:54.673014 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d903116_17f2_4664_8e0e_6499258fab01.slice/crio-93f7d0608eeef2db870f464d2794d22a6c04e0fb67865fd18a494eb36d66cbfe WatchSource:0}: Error finding container 93f7d0608eeef2db870f464d2794d22a6c04e0fb67865fd18a494eb36d66cbfe: Status 404 returned error can't find the container with id 93f7d0608eeef2db870f464d2794d22a6c04e0fb67865fd18a494eb36d66cbfe Jul 17 21:32:54.674921 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:54.674907 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:32:55.625969 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:55.625896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" event={"ID":"0d903116-17f2-4664-8e0e-6499258fab01","Type":"ContainerStarted","Data":"93f7d0608eeef2db870f464d2794d22a6c04e0fb67865fd18a494eb36d66cbfe"} Jul 17 21:32:57.635904 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:57.635815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" event={"ID":"0d903116-17f2-4664-8e0e-6499258fab01","Type":"ContainerStarted","Data":"ed0014e025ea859a9f8211156b7d0593427f2c92d442b7dc151f0d047e24ba70"} Jul 17 21:32:57.661321 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:32:57.661255 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-788f6fc9d-5mrmr" podStartSLOduration=0.991344427 podStartE2EDuration="3.661215072s" totalImagesPullingTime="2.669870645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:32:57.659656967 +0000 UTC m=+314.929654671" watchObservedRunningTime="2026-07-17 21:32:57.661215072 +0000 UTC m=+314.931212776" Jul 17 21:33:01.245589 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.245554 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d"] Jul 17 21:33:01.248994 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.248972 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.252782 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.252761 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-w8q4p\"" Jul 17 21:33:01.253256 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.253219 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 21:33:01.260571 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.260550 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d"] Jul 17 21:33:01.362438 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362438 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362436 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362458 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362501 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362610 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362632 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kbn\" (UniqueName: \"kubernetes.io/projected/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-kube-api-access-r8kbn\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.362823 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.362678 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463371 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463371 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463452 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463485 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kbn\" (UniqueName: \"kubernetes.io/projected/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-kube-api-access-r8kbn\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463573 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.463922 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463856 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.464018 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.463998 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.464087 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.464029 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.464184 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.464164 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.464262 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.464185 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.466009 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.465985 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.466195 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.466177 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.483480 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.483448 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.483597 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.483510 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kbn\" (UniqueName: \"kubernetes.io/projected/7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9-kube-api-access-r8kbn\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678wmx4d\" (UID: \"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.559701 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.559611 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:01.702272 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:01.702224 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d"] Jul 17 21:33:01.705525 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:33:01.705498 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5f1fbc_c75c_4d00_a1ba_4f1aa073e3e9.slice/crio-2066f67e774f0c3fc7f5327235282de94776789f69f94b00b666e8e4deaf5eba WatchSource:0}: Error finding container 2066f67e774f0c3fc7f5327235282de94776789f69f94b00b666e8e4deaf5eba: Status 404 returned error can't find the container with id 2066f67e774f0c3fc7f5327235282de94776789f69f94b00b666e8e4deaf5eba Jul 17 21:33:02.657161 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:02.657125 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" event={"ID":"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9","Type":"ContainerStarted","Data":"2066f67e774f0c3fc7f5327235282de94776789f69f94b00b666e8e4deaf5eba"} Jul 17 21:33:03.476011 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.475728 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c"] Jul 17 21:33:03.483774 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.483355 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.486347 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.486322 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 21:33:03.486469 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.486390 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kqn6x\"" Jul 17 21:33:03.486690 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.486673 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 21:33:03.489264 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.489058 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 21:33:03.499292 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.499263 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c"] Jul 17 21:33:03.585040 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.585004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.585293 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.585077 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55266\" (UniqueName: \"kubernetes.io/projected/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-kube-api-access-55266\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.585293 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.585133 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.585293 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.585164 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.585293 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.585213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.585293 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.585284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.585670 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.585307 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.686440 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.686400 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55266\" (UniqueName: \"kubernetes.io/projected/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-kube-api-access-55266\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.686918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.686451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.686918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.686472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.686918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.686504 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.686918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.686544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.686918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.686571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.686918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.686634 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.687426 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.687352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.689638 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.689612 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.689791 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.689756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.689880 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.689829 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.690295 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.690270 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.706825 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.706797 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55266\" (UniqueName: \"kubernetes.io/projected/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-kube-api-access-55266\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.708486 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.708452 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b96ade98-d3a6-41e7-b78b-3f50d11f9df6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cv98c\" (UID: \"b96ade98-d3a6-41e7-b78b-3f50d11f9df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:03.801302 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:03.800615 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:04.431004 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:04.430952 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:33:04.431100 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:04.431071 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:33:04.431159 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:04.431114 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:33:04.506036 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:04.506010 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c"] Jul 17 21:33:04.508667 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:33:04.508639 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ade98_d3a6_41e7_b78b_3f50d11f9df6.slice/crio-acf73d99cc175d2c140c4e5885ad1010d22fd17a3d2ed2bedaa9b7b10bcad08a WatchSource:0}: Error finding container acf73d99cc175d2c140c4e5885ad1010d22fd17a3d2ed2bedaa9b7b10bcad08a: Status 404 returned error can't find the container with id acf73d99cc175d2c140c4e5885ad1010d22fd17a3d2ed2bedaa9b7b10bcad08a Jul 17 21:33:04.667659 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:04.667625 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" event={"ID":"7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9","Type":"ContainerStarted","Data":"b3832aebb9db4af8b3ab5d47b2a9658546ab8177158effeff818b3cbf714e705"} Jul 17 21:33:04.668912 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:04.668883 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" event={"ID":"b96ade98-d3a6-41e7-b78b-3f50d11f9df6","Type":"ContainerStarted","Data":"acf73d99cc175d2c140c4e5885ad1010d22fd17a3d2ed2bedaa9b7b10bcad08a"} Jul 17 21:33:04.707116 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:04.707059 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" podStartSLOduration=0.984052091 podStartE2EDuration="3.707039028s" totalImagesPullingTime="2.722986937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:04.703839135 +0000 UTC m=+321.973836841" watchObservedRunningTime="2026-07-17 21:33:04.707039028 +0000 UTC m=+321.977036733" Jul 17 21:33:05.560810 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.560774 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:05.566671 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.566642 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:05.673679 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.673609 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:05.674992 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.674965 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678wmx4d" Jul 17 21:33:05.974418 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.974323 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g"] Jul 17 21:33:05.978177 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.978086 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:05.981197 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.981175 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 17 21:33:05.981570 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.981555 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-568br\"" Jul 17 21:33:05.993964 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:05.993937 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g"] Jul 17 21:33:06.109869 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:06.109833 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:06.110058 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:06.109904 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lgs\" (UniqueName: \"kubernetes.io/projected/e9da3247-ff20-4211-ad5f-640426eb1a6d-kube-api-access-59lgs\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:06.211042 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:06.211004 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:06.211212 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:06.211070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lgs\" (UniqueName: \"kubernetes.io/projected/e9da3247-ff20-4211-ad5f-640426eb1a6d-kube-api-access-59lgs\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:06.211319 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:33:06.211297 2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 17 21:33:06.211388 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:33:06.211376 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert podName:e9da3247-ff20-4211-ad5f-640426eb1a6d nodeName:}" failed. No retries permitted until 2026-07-17 21:33:06.711352903 +0000 UTC m=+323.981350589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert") pod "llmisvc-controller-manager-777c6bd7d5-m5p9g" (UID: "e9da3247-ff20-4211-ad5f-640426eb1a6d") : secret "llmisvc-webhook-server-cert" not found Jul 17 21:33:06.225785 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:06.225718 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lgs\" (UniqueName: \"kubernetes.io/projected/e9da3247-ff20-4211-ad5f-640426eb1a6d-kube-api-access-59lgs\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:06.716144 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:06.716107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:06.716359 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:33:06.716298 2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 17 21:33:06.716430 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:33:06.716366 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert podName:e9da3247-ff20-4211-ad5f-640426eb1a6d nodeName:}" failed. No retries permitted until 2026-07-17 21:33:07.716347994 +0000 UTC m=+324.986345687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert") pod "llmisvc-controller-manager-777c6bd7d5-m5p9g" (UID: "e9da3247-ff20-4211-ad5f-640426eb1a6d") : secret "llmisvc-webhook-server-cert" not found Jul 17 21:33:07.183391 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.183352 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:33:07.183665 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.183419 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:33:07.688470 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.688421 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" event={"ID":"b96ade98-d3a6-41e7-b78b-3f50d11f9df6","Type":"ContainerStarted","Data":"eb902fc72992fb34b6cd0352114d602c4cb571ac658606ebaf3828eaf83ab01f"} Jul 17 21:33:07.688687 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.688646 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:07.712588 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.712524 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" podStartSLOduration=2.040257572 podStartE2EDuration="4.712501571s" totalImagesPullingTime="2.672243999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:07.710583914 +0000 UTC m=+324.980581645" watchObservedRunningTime="2026-07-17 21:33:07.712501571 +0000 UTC m=+324.982499277" Jul 17 21:33:07.730590 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.730552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:07.733284 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.733222 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9da3247-ff20-4211-ad5f-640426eb1a6d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-m5p9g\" (UID: \"e9da3247-ff20-4211-ad5f-640426eb1a6d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:07.791361 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.791328 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:07.957996 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:07.957914 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g"] Jul 17 21:33:07.961641 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:33:07.961613 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode9da3247_ff20_4211_ad5f_640426eb1a6d.slice/crio-8774fb8a1fbccd1d15871e8a8885ff100f74d457bba03cda534128cdf0db786d WatchSource:0}: Error finding container 8774fb8a1fbccd1d15871e8a8885ff100f74d457bba03cda534128cdf0db786d: Status 404 returned error can't find the container with id 8774fb8a1fbccd1d15871e8a8885ff100f74d457bba03cda534128cdf0db786d Jul 17 21:33:08.695794 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:08.695753 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" event={"ID":"e9da3247-ff20-4211-ad5f-640426eb1a6d","Type":"ContainerStarted","Data":"8774fb8a1fbccd1d15871e8a8885ff100f74d457bba03cda534128cdf0db786d"} Jul 17 21:33:08.697826 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:08.697805 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cv98c" Jul 17 21:33:11.715556 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:11.715517 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" event={"ID":"e9da3247-ff20-4211-ad5f-640426eb1a6d","Type":"ContainerStarted","Data":"22f25f7a806c1f1f60f94d7f6ac9f6fc3cd2bbecced9535e0dbc0c421c569637"} Jul 17 21:33:11.716010 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:11.715598 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:11.733022 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:11.732976 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" podStartSLOduration=3.812337274 podStartE2EDuration="6.732960825s" totalImagesPullingTime="2.920623551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:11.731897215 +0000 UTC m=+329.001894916" watchObservedRunningTime="2026-07-17 21:33:11.732960825 +0000 UTC m=+329.002958552" Jul 17 21:33:42.721994 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:42.721958 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-m5p9g" Jul 17 21:33:55.302883 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.302852 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zt6qr"] Jul 17 21:33:55.311491 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.311463 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" Jul 17 21:33:55.314042 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.314018 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 21:33:55.314180 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.314018 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-t2xg8\"" Jul 17 21:33:55.315095 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.315074 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 21:33:55.317751 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.317730 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zt6qr"] Jul 17 21:33:55.446618 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.446579 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzztp\" (UniqueName: \"kubernetes.io/projected/9faca16a-1a61-48cf-aed7-e522498f03c9-kube-api-access-lzztp\") pod \"kuadrant-operator-catalog-zt6qr\" (UID: \"9faca16a-1a61-48cf-aed7-e522498f03c9\") " pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" Jul 17 21:33:55.548171 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.548136 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzztp\" (UniqueName: \"kubernetes.io/projected/9faca16a-1a61-48cf-aed7-e522498f03c9-kube-api-access-lzztp\") pod \"kuadrant-operator-catalog-zt6qr\" (UID: \"9faca16a-1a61-48cf-aed7-e522498f03c9\") " pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" Jul 17 21:33:55.557598 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.557531 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzztp\" (UniqueName: \"kubernetes.io/projected/9faca16a-1a61-48cf-aed7-e522498f03c9-kube-api-access-lzztp\") pod \"kuadrant-operator-catalog-zt6qr\" (UID: \"9faca16a-1a61-48cf-aed7-e522498f03c9\") " pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" Jul 17 21:33:55.623135 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.623103 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" Jul 17 21:33:55.676386 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.676353 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zt6qr"] Jul 17 21:33:55.759706 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.759686 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zt6qr"] Jul 17 21:33:55.761452 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:33:55.761418 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faca16a_1a61_48cf_aed7_e522498f03c9.slice/crio-a2ae278cbad252631dfcd297f59692b05e6990808c0ba9b9f379c4cb1d03a9fe WatchSource:0}: Error finding container a2ae278cbad252631dfcd297f59692b05e6990808c0ba9b9f379c4cb1d03a9fe: Status 404 returned error can't find the container with id a2ae278cbad252631dfcd297f59692b05e6990808c0ba9b9f379c4cb1d03a9fe Jul 17 21:33:55.879084 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.879008 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njq74"] Jul 17 21:33:55.881516 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.881499 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:33:55.890167 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.890142 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njq74"] Jul 17 21:33:55.907779 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.907740 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" event={"ID":"9faca16a-1a61-48cf-aed7-e522498f03c9","Type":"ContainerStarted","Data":"a2ae278cbad252631dfcd297f59692b05e6990808c0ba9b9f379c4cb1d03a9fe"} Jul 17 21:33:55.952110 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:55.952074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n56q\" (UniqueName: \"kubernetes.io/projected/8743d429-dd45-4cac-a688-8e976e5fd43e-kube-api-access-8n56q\") pod \"kuadrant-operator-catalog-njq74\" (UID: \"8743d429-dd45-4cac-a688-8e976e5fd43e\") " pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:33:56.052712 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:56.052663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n56q\" (UniqueName: \"kubernetes.io/projected/8743d429-dd45-4cac-a688-8e976e5fd43e-kube-api-access-8n56q\") pod \"kuadrant-operator-catalog-njq74\" (UID: \"8743d429-dd45-4cac-a688-8e976e5fd43e\") " pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:33:56.062039 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:56.062002 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n56q\" (UniqueName: \"kubernetes.io/projected/8743d429-dd45-4cac-a688-8e976e5fd43e-kube-api-access-8n56q\") pod \"kuadrant-operator-catalog-njq74\" (UID: \"8743d429-dd45-4cac-a688-8e976e5fd43e\") " pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:33:56.191725 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:56.191693 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:33:56.323426 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:56.323394 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-njq74"] Jul 17 21:33:56.325968 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:33:56.325936 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8743d429_dd45_4cac_a688_8e976e5fd43e.slice/crio-3ee0ff809907e6a9901f9b6d8c9b84d038a832767c2670d461aab78bf14b6501 WatchSource:0}: Error finding container 3ee0ff809907e6a9901f9b6d8c9b84d038a832767c2670d461aab78bf14b6501: Status 404 returned error can't find the container with id 3ee0ff809907e6a9901f9b6d8c9b84d038a832767c2670d461aab78bf14b6501 Jul 17 21:33:56.915697 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:56.915658 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-njq74" event={"ID":"8743d429-dd45-4cac-a688-8e976e5fd43e","Type":"ContainerStarted","Data":"3ee0ff809907e6a9901f9b6d8c9b84d038a832767c2670d461aab78bf14b6501"} Jul 17 21:33:57.923050 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:57.923012 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-njq74" event={"ID":"8743d429-dd45-4cac-a688-8e976e5fd43e","Type":"ContainerStarted","Data":"17ec70e04b1af7342068106b87c7c9226bb2ada3f47e0bfdef8323543da0d534"} Jul 17 21:33:57.924775 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:57.924748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" event={"ID":"9faca16a-1a61-48cf-aed7-e522498f03c9","Type":"ContainerStarted","Data":"3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345"} Jul 17 21:33:57.924899 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:57.924833 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" podUID="9faca16a-1a61-48cf-aed7-e522498f03c9" containerName="registry-server" containerID="cri-o://3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345" gracePeriod=2 Jul 17 21:33:57.941142 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:57.941083 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-njq74" podStartSLOduration=1.442200785 podStartE2EDuration="2.941064816s" totalImagesPullingTime="1.498864031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:57.939042829 +0000 UTC m=+375.209040556" watchObservedRunningTime="2026-07-17 21:33:57.941064816 +0000 UTC m=+375.211062555" Jul 17 21:33:57.954996 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:57.954919 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" podStartSLOduration=0.915109957 podStartE2EDuration="2.954896271s" totalImagesPullingTime="2.039786314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:33:57.954145561 +0000 UTC m=+375.224143268" watchObservedRunningTime="2026-07-17 21:33:57.954896271 +0000 UTC m=+375.224893976" Jul 17 21:33:58.167750 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.167722 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zt6qr_9faca16a-1a61-48cf-aed7-e522498f03c9/registry-server/0.log" Jul 17 21:33:58.169009 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.168994 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" Jul 17 21:33:58.273151 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.273113 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzztp\" (UniqueName: \"kubernetes.io/projected/9faca16a-1a61-48cf-aed7-e522498f03c9-kube-api-access-lzztp\") pod \"9faca16a-1a61-48cf-aed7-e522498f03c9\" (UID: \"9faca16a-1a61-48cf-aed7-e522498f03c9\") " Jul 17 21:33:58.275522 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.275486 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9faca16a-1a61-48cf-aed7-e522498f03c9-kube-api-access-lzztp" (OuterVolumeSpecName: "kube-api-access-lzztp") pod "9faca16a-1a61-48cf-aed7-e522498f03c9" (UID: "9faca16a-1a61-48cf-aed7-e522498f03c9"). InnerVolumeSpecName "kube-api-access-lzztp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:33:58.374504 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.374408 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzztp\" (UniqueName: \"kubernetes.io/projected/9faca16a-1a61-48cf-aed7-e522498f03c9-kube-api-access-lzztp\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:33:58.930652 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.930624 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zt6qr_9faca16a-1a61-48cf-aed7-e522498f03c9/registry-server/0.log" Jul 17 21:33:58.931056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.930661 2566 generic.go:356] "Generic (PLEG): container finished" podID="9faca16a-1a61-48cf-aed7-e522498f03c9" containerID="3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345" exitCode=1 Jul 17 21:33:58.931056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.930696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" event={"ID":"9faca16a-1a61-48cf-aed7-e522498f03c9","Type":"ContainerDied","Data":"3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345"} Jul 17 21:33:58.931056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.930721 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" Jul 17 21:33:58.931056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.930761 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zt6qr" event={"ID":"9faca16a-1a61-48cf-aed7-e522498f03c9","Type":"ContainerDied","Data":"a2ae278cbad252631dfcd297f59692b05e6990808c0ba9b9f379c4cb1d03a9fe"} Jul 17 21:33:58.931056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.930800 2566 scope.go:117] "RemoveContainer" containerID="3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345" Jul 17 21:33:58.939918 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.939900 2566 scope.go:117] "RemoveContainer" containerID="3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345" Jul 17 21:33:58.940158 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:33:58.940140 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345\": container with ID starting with 3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345 not found: ID does not exist" containerID="3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345" Jul 17 21:33:58.940219 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.940167 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345"} err="failed to get container status \"3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345\": rpc error: code = NotFound desc = could not find container \"3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345\": container with ID starting with 3af08a70477a76453ff54d8fcb06f3a2ed819ce4e7112cf612bd6adf6e056345 not found: ID does not exist" Jul 17 21:33:58.951073 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.951049 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zt6qr"] Jul 17 21:33:58.955069 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:58.955039 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zt6qr"] Jul 17 21:33:59.298047 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:33:59.298016 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9faca16a-1a61-48cf-aed7-e522498f03c9" path="/var/lib/kubelet/pods/9faca16a-1a61-48cf-aed7-e522498f03c9/volumes" Jul 17 21:34:06.191965 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:06.191919 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:34:06.192528 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:06.191990 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:34:06.214936 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:06.214907 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:34:06.988275 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:06.988247 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-njq74" Jul 17 21:34:22.552688 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.552653 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9"] Jul 17 21:34:22.553073 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.553002 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faca16a-1a61-48cf-aed7-e522498f03c9" containerName="registry-server" Jul 17 21:34:22.553073 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.553013 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faca16a-1a61-48cf-aed7-e522498f03c9" containerName="registry-server" Jul 17 21:34:22.553073 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.553069 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faca16a-1a61-48cf-aed7-e522498f03c9" containerName="registry-server" Jul 17 21:34:22.556010 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.555993 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:22.558473 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.558452 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6vzr6\"" Jul 17 21:34:22.563937 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.563916 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9"] Jul 17 21:34:22.583889 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.583854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vpj\" (UniqueName: \"kubernetes.io/projected/29321f7d-d3fe-4fc0-8ed5-762457550031-kube-api-access-n4vpj\") pod \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" (UID: \"29321f7d-d3fe-4fc0-8ed5-762457550031\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:22.684489 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.684449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vpj\" (UniqueName: \"kubernetes.io/projected/29321f7d-d3fe-4fc0-8ed5-762457550031-kube-api-access-n4vpj\") pod \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" (UID: \"29321f7d-d3fe-4fc0-8ed5-762457550031\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:22.706502 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.706475 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vpj\" (UniqueName: \"kubernetes.io/projected/29321f7d-d3fe-4fc0-8ed5-762457550031-kube-api-access-n4vpj\") pod \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" (UID: \"29321f7d-d3fe-4fc0-8ed5-762457550031\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:22.866771 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.866663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:22.998266 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:22.998214 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9"] Jul 17 21:34:23.002458 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:34:23.002420 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29321f7d_d3fe_4fc0_8ed5_762457550031.slice/crio-6ffe0102677a9013aefc3c7386d8c6a63942e917c0b03fe53d2b468499f24bea WatchSource:0}: Error finding container 6ffe0102677a9013aefc3c7386d8c6a63942e917c0b03fe53d2b468499f24bea: Status 404 returned error can't find the container with id 6ffe0102677a9013aefc3c7386d8c6a63942e917c0b03fe53d2b468499f24bea Jul 17 21:34:23.035719 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:23.035686 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" event={"ID":"29321f7d-d3fe-4fc0-8ed5-762457550031","Type":"ContainerStarted","Data":"6ffe0102677a9013aefc3c7386d8c6a63942e917c0b03fe53d2b468499f24bea"} Jul 17 21:34:24.545045 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:24.545011 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d4ffc6fb-gvgsp"] Jul 17 21:34:25.047631 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:25.047591 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" event={"ID":"29321f7d-d3fe-4fc0-8ed5-762457550031","Type":"ContainerStarted","Data":"9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9"} Jul 17 21:34:25.047824 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:25.047721 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:25.066790 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:25.066728 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" podStartSLOduration=1.688265143 podStartE2EDuration="3.066711238s" totalImagesPullingTime="1.378446095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:25.065425276 +0000 UTC m=+402.335422981" watchObservedRunningTime="2026-07-17 21:34:25.066711238 +0000 UTC m=+402.336708942" Jul 17 21:34:30.528032 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.528001 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd"] Jul 17 21:34:30.534899 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.534881 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" Jul 17 21:34:30.537701 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.537684 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 17 21:34:30.537885 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.537865 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6kdss\"" Jul 17 21:34:30.545894 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.545873 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd"] Jul 17 21:34:30.656058 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.656019 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clz2j\" (UniqueName: \"kubernetes.io/projected/c593492f-d381-4054-9845-794427bd2bde-kube-api-access-clz2j\") pod \"dns-operator-controller-manager-864dd78c4c-pdnfd\" (UID: \"c593492f-d381-4054-9845-794427bd2bde\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" Jul 17 21:34:30.757020 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.756982 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clz2j\" (UniqueName: \"kubernetes.io/projected/c593492f-d381-4054-9845-794427bd2bde-kube-api-access-clz2j\") pod \"dns-operator-controller-manager-864dd78c4c-pdnfd\" (UID: \"c593492f-d381-4054-9845-794427bd2bde\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" Jul 17 21:34:30.765764 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.765729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clz2j\" (UniqueName: \"kubernetes.io/projected/c593492f-d381-4054-9845-794427bd2bde-kube-api-access-clz2j\") pod \"dns-operator-controller-manager-864dd78c4c-pdnfd\" (UID: \"c593492f-d381-4054-9845-794427bd2bde\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" Jul 17 21:34:30.845839 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:30.845735 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" Jul 17 21:34:31.022249 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:31.022202 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd"] Jul 17 21:34:31.024309 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:34:31.024280 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc593492f_d381_4054_9845_794427bd2bde.slice/crio-b285417dd01ad995add69d2b4fc80f80f472b17497fc1daf51c37363cd877a9f WatchSource:0}: Error finding container b285417dd01ad995add69d2b4fc80f80f472b17497fc1daf51c37363cd877a9f: Status 404 returned error can't find the container with id b285417dd01ad995add69d2b4fc80f80f472b17497fc1daf51c37363cd877a9f Jul 17 21:34:31.087049 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:31.087018 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" event={"ID":"c593492f-d381-4054-9845-794427bd2bde","Type":"ContainerStarted","Data":"b285417dd01ad995add69d2b4fc80f80f472b17497fc1daf51c37363cd877a9f"} Jul 17 21:34:33.099408 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:33.099378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" event={"ID":"c593492f-d381-4054-9845-794427bd2bde","Type":"ContainerStarted","Data":"048a332e2bb3d863c7877861532d33cad1030b278a7d538006bab4e044b44871"} Jul 17 21:34:33.099802 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:33.099469 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" Jul 17 21:34:33.119143 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:33.119036 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" podStartSLOduration=1.218370241 podStartE2EDuration="3.119016826s" totalImagesPullingTime="1.900646585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:33.118257148 +0000 UTC m=+410.388254871" watchObservedRunningTime="2026-07-17 21:34:33.119016826 +0000 UTC m=+410.389014530" Jul 17 21:34:34.361590 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.361556 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp"] Jul 17 21:34:34.364790 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.364774 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" Jul 17 21:34:34.367473 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.367454 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xfbm5\"" Jul 17 21:34:34.379130 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.379105 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp"] Jul 17 21:34:34.492384 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.492339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkd9\" (UniqueName: \"kubernetes.io/projected/c9f38acd-80dd-4365-9cfa-05cc86f18150-kube-api-access-2zkd9\") pod \"authorino-operator-75cc8d5cb7-mq5jp\" (UID: \"c9f38acd-80dd-4365-9cfa-05cc86f18150\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" Jul 17 21:34:34.593632 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.593593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkd9\" (UniqueName: \"kubernetes.io/projected/c9f38acd-80dd-4365-9cfa-05cc86f18150-kube-api-access-2zkd9\") pod \"authorino-operator-75cc8d5cb7-mq5jp\" (UID: \"c9f38acd-80dd-4365-9cfa-05cc86f18150\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" Jul 17 21:34:34.602196 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.602169 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkd9\" (UniqueName: \"kubernetes.io/projected/c9f38acd-80dd-4365-9cfa-05cc86f18150-kube-api-access-2zkd9\") pod \"authorino-operator-75cc8d5cb7-mq5jp\" (UID: \"c9f38acd-80dd-4365-9cfa-05cc86f18150\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" Jul 17 21:34:34.674897 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.674866 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" Jul 17 21:34:34.816809 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:34.816784 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp"] Jul 17 21:34:34.818927 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:34:34.818895 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f38acd_80dd_4365_9cfa_05cc86f18150.slice/crio-a4b30c4a9269460c9443ab75f1c4fe145a2fa76795e8e7f971d381cb498aac79 WatchSource:0}: Error finding container a4b30c4a9269460c9443ab75f1c4fe145a2fa76795e8e7f971d381cb498aac79: Status 404 returned error can't find the container with id a4b30c4a9269460c9443ab75f1c4fe145a2fa76795e8e7f971d381cb498aac79 Jul 17 21:34:35.108965 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:35.108873 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" event={"ID":"c9f38acd-80dd-4365-9cfa-05cc86f18150","Type":"ContainerStarted","Data":"a4b30c4a9269460c9443ab75f1c4fe145a2fa76795e8e7f971d381cb498aac79"} Jul 17 21:34:36.054988 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:36.054955 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:38.124481 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:38.124449 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" event={"ID":"c9f38acd-80dd-4365-9cfa-05cc86f18150","Type":"ContainerStarted","Data":"ca4d1cbb8330ac93e1bafdc2d8955665ced0e56da19be19bac363cfa6b79bc60"} Jul 17 21:34:38.124926 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:38.124570 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" Jul 17 21:34:38.142254 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:38.142195 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" podStartSLOduration=1.853001017 podStartE2EDuration="4.142179929s" totalImagesPullingTime="2.289178912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:38.140159925 +0000 UTC m=+415.410157652" watchObservedRunningTime="2026-07-17 21:34:38.142179929 +0000 UTC m=+415.412177632" Jul 17 21:34:44.105856 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:44.105827 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-pdnfd" Jul 17 21:34:47.458856 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:47.458809 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9"] Jul 17 21:34:47.459330 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:47.459304 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" containerName="manager" containerID="cri-o://9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9" gracePeriod=2 Jul 17 21:34:47.468574 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:47.468547 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9"] Jul 17 21:34:47.694181 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:47.694155 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:47.696530 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:34:47.696499 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" is forbidden: User \"system:node:ip-10-0-139-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-178.ec2.internal' and this object" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:47.820029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:47.819919 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4vpj\" (UniqueName: \"kubernetes.io/projected/29321f7d-d3fe-4fc0-8ed5-762457550031-kube-api-access-n4vpj\") pod \"29321f7d-d3fe-4fc0-8ed5-762457550031\" (UID: \"29321f7d-d3fe-4fc0-8ed5-762457550031\") " Jul 17 21:34:47.822110 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:47.822087 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29321f7d-d3fe-4fc0-8ed5-762457550031-kube-api-access-n4vpj" (OuterVolumeSpecName: "kube-api-access-n4vpj") pod "29321f7d-d3fe-4fc0-8ed5-762457550031" (UID: "29321f7d-d3fe-4fc0-8ed5-762457550031"). InnerVolumeSpecName "kube-api-access-n4vpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:34:47.921219 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:47.921180 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vpj\" (UniqueName: \"kubernetes.io/projected/29321f7d-d3fe-4fc0-8ed5-762457550031-kube-api-access-n4vpj\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:48.180939 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.180909 2566 generic.go:356] "Generic (PLEG): container finished" podID="29321f7d-d3fe-4fc0-8ed5-762457550031" containerID="9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9" exitCode=0 Jul 17 21:34:48.181126 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.180954 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:48.181126 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.181022 2566 scope.go:117] "RemoveContainer" containerID="9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9" Jul 17 21:34:48.183487 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:34:48.183459 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" is forbidden: User \"system:node:ip-10-0-139-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-178.ec2.internal' and this object" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:48.190073 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.190059 2566 scope.go:117] "RemoveContainer" containerID="9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9" Jul 17 21:34:48.190355 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:34:48.190338 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9\": container with ID starting with 9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9 not found: ID does not exist" containerID="9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9" Jul 17 21:34:48.190426 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.190362 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9"} err="failed to get container status \"9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9\": rpc error: code = NotFound desc = could not find container \"9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9\": container with ID starting with 9e786858aa732556d2cbac7e1fe57d095d2a12201c975dc789211320f97184a9 not found: ID does not exist" Jul 17 21:34:48.191075 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:34:48.191056 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" is forbidden: User \"system:node:ip-10-0-139-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-178.ec2.internal' and this object" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:48.411606 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.411572 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc"] Jul 17 21:34:48.411898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.411885 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" containerName="manager" Jul 17 21:34:48.411991 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.411899 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" containerName="manager" Jul 17 21:34:48.411991 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.411964 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" containerName="manager" Jul 17 21:34:48.416275 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.416223 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.419212 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.419193 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhf9w\"" Jul 17 21:34:48.431435 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.431377 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc"] Jul 17 21:34:48.451740 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:34:48.451702 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" is forbidden: User \"system:node:ip-10-0-139-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-178.ec2.internal' and this object" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:48.526792 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.526752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f27bbac-968e-4765-9080-05b85d408e25-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9hcbc\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.527139 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.526836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgxl\" (UniqueName: \"kubernetes.io/projected/4f27bbac-968e-4765-9080-05b85d408e25-kube-api-access-vwgxl\") pod \"kuadrant-operator-controller-manager-86878c769f-9hcbc\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.627318 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.627280 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgxl\" (UniqueName: \"kubernetes.io/projected/4f27bbac-968e-4765-9080-05b85d408e25-kube-api-access-vwgxl\") pod \"kuadrant-operator-controller-manager-86878c769f-9hcbc\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.627489 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.627357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f27bbac-968e-4765-9080-05b85d408e25-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9hcbc\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.627686 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.627671 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f27bbac-968e-4765-9080-05b85d408e25-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9hcbc\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.638103 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.638072 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgxl\" (UniqueName: \"kubernetes.io/projected/4f27bbac-968e-4765-9080-05b85d408e25-kube-api-access-vwgxl\") pod \"kuadrant-operator-controller-manager-86878c769f-9hcbc\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.726221 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.726136 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:48.864796 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:48.864760 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc"] Jul 17 21:34:48.868482 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:34:48.868450 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f27bbac_968e_4765_9080_05b85d408e25.slice/crio-3fccc00d2075986b682a4ed7592daadebef0cee3bb08aead57ce44a101298c04 WatchSource:0}: Error finding container 3fccc00d2075986b682a4ed7592daadebef0cee3bb08aead57ce44a101298c04: Status 404 returned error can't find the container with id 3fccc00d2075986b682a4ed7592daadebef0cee3bb08aead57ce44a101298c04 Jul 17 21:34:49.131518 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.131431 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mq5jp" Jul 17 21:34:49.133772 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:34:49.133737 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mh2v9\" is forbidden: User \"system:node:ip-10-0-139-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-178.ec2.internal' and this object" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mh2v9" Jul 17 21:34:49.187560 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.187529 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" event={"ID":"4f27bbac-968e-4765-9080-05b85d408e25","Type":"ContainerStarted","Data":"3fccc00d2075986b682a4ed7592daadebef0cee3bb08aead57ce44a101298c04"} Jul 17 21:34:49.300000 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.299951 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29321f7d-d3fe-4fc0-8ed5-762457550031" path="/var/lib/kubelet/pods/29321f7d-d3fe-4fc0-8ed5-762457550031/volumes" Jul 17 21:34:49.574310 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.574221 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57d4ffc6fb-gvgsp" podUID="219b3cd0-0636-4ecd-85a7-06976b5b2900" containerName="console" containerID="cri-o://c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024" gracePeriod=15 Jul 17 21:34:49.830029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.829964 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:34:49.939263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939210 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-serving-cert\") pod \"219b3cd0-0636-4ecd-85a7-06976b5b2900\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " Jul 17 21:34:49.939263 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939266 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wf86\" (UniqueName: \"kubernetes.io/projected/219b3cd0-0636-4ecd-85a7-06976b5b2900-kube-api-access-5wf86\") pod \"219b3cd0-0636-4ecd-85a7-06976b5b2900\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " Jul 17 21:34:49.939510 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939309 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-oauth-serving-cert\") pod \"219b3cd0-0636-4ecd-85a7-06976b5b2900\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " Jul 17 21:34:49.939510 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939329 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-service-ca\") pod \"219b3cd0-0636-4ecd-85a7-06976b5b2900\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " Jul 17 21:34:49.939510 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-oauth-config\") pod \"219b3cd0-0636-4ecd-85a7-06976b5b2900\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " Jul 17 21:34:49.939510 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939442 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-config\") pod \"219b3cd0-0636-4ecd-85a7-06976b5b2900\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " Jul 17 21:34:49.939510 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939484 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-trusted-ca-bundle\") pod \"219b3cd0-0636-4ecd-85a7-06976b5b2900\" (UID: \"219b3cd0-0636-4ecd-85a7-06976b5b2900\") " Jul 17 21:34:49.939785 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.939745 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "219b3cd0-0636-4ecd-85a7-06976b5b2900" (UID: "219b3cd0-0636-4ecd-85a7-06976b5b2900"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:34:49.940068 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.940040 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "219b3cd0-0636-4ecd-85a7-06976b5b2900" (UID: "219b3cd0-0636-4ecd-85a7-06976b5b2900"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:34:49.940068 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.940036 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-config" (OuterVolumeSpecName: "console-config") pod "219b3cd0-0636-4ecd-85a7-06976b5b2900" (UID: "219b3cd0-0636-4ecd-85a7-06976b5b2900"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:34:49.940266 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.940192 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-service-ca" (OuterVolumeSpecName: "service-ca") pod "219b3cd0-0636-4ecd-85a7-06976b5b2900" (UID: "219b3cd0-0636-4ecd-85a7-06976b5b2900"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:34:49.941873 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.941848 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "219b3cd0-0636-4ecd-85a7-06976b5b2900" (UID: "219b3cd0-0636-4ecd-85a7-06976b5b2900"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:34:49.942113 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.942092 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "219b3cd0-0636-4ecd-85a7-06976b5b2900" (UID: "219b3cd0-0636-4ecd-85a7-06976b5b2900"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:34:49.942197 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:49.942181 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219b3cd0-0636-4ecd-85a7-06976b5b2900-kube-api-access-5wf86" (OuterVolumeSpecName: "kube-api-access-5wf86") pod "219b3cd0-0636-4ecd-85a7-06976b5b2900" (UID: "219b3cd0-0636-4ecd-85a7-06976b5b2900"). InnerVolumeSpecName "kube-api-access-5wf86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:34:50.040840 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.040803 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-oauth-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:50.040840 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.040831 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-config\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:50.040840 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.040846 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-trusted-ca-bundle\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:50.041101 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.040861 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/219b3cd0-0636-4ecd-85a7-06976b5b2900-console-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:50.041101 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.040871 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wf86\" (UniqueName: \"kubernetes.io/projected/219b3cd0-0636-4ecd-85a7-06976b5b2900-kube-api-access-5wf86\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:50.041101 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.040879 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-oauth-serving-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:50.041101 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.040888 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/219b3cd0-0636-4ecd-85a7-06976b5b2900-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:34:50.195007 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.194968 2566 generic.go:356] "Generic (PLEG): container finished" podID="219b3cd0-0636-4ecd-85a7-06976b5b2900" containerID="c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024" exitCode=0 Jul 17 21:34:50.195199 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.195024 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d4ffc6fb-gvgsp" Jul 17 21:34:50.195199 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.195038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d4ffc6fb-gvgsp" event={"ID":"219b3cd0-0636-4ecd-85a7-06976b5b2900","Type":"ContainerDied","Data":"c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024"} Jul 17 21:34:50.195199 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.195089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d4ffc6fb-gvgsp" event={"ID":"219b3cd0-0636-4ecd-85a7-06976b5b2900","Type":"ContainerDied","Data":"b283e4cc035aec591a5060ec661239b3c08242772caae58b1e868807d86e9629"} Jul 17 21:34:50.195199 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.195116 2566 scope.go:117] "RemoveContainer" containerID="c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024" Jul 17 21:34:50.205939 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.205918 2566 scope.go:117] "RemoveContainer" containerID="c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024" Jul 17 21:34:50.206273 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:34:50.206250 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024\": container with ID starting with c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024 not found: ID does not exist" containerID="c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024" Jul 17 21:34:50.206365 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.206284 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024"} err="failed to get container status \"c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024\": rpc error: code = NotFound desc = could not find container \"c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024\": container with ID starting with c8e91ea2b22ce1cbbd5c16a9d281febfe8a5b04e93b54abe0380fec6080ae024 not found: ID does not exist" Jul 17 21:34:50.241704 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.241673 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d4ffc6fb-gvgsp"] Jul 17 21:34:50.245614 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:50.245558 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d4ffc6fb-gvgsp"] Jul 17 21:34:51.298655 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:51.298621 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219b3cd0-0636-4ecd-85a7-06976b5b2900" path="/var/lib/kubelet/pods/219b3cd0-0636-4ecd-85a7-06976b5b2900/volumes" Jul 17 21:34:53.212281 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:53.212245 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" event={"ID":"4f27bbac-968e-4765-9080-05b85d408e25","Type":"ContainerStarted","Data":"f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f"} Jul 17 21:34:53.212727 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:53.212401 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:34:53.234364 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:34:53.234316 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" podStartSLOduration=1.305340219 podStartE2EDuration="5.234299811s" totalImagesPullingTime="3.928959592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:34:53.231735814 +0000 UTC m=+430.501733550" watchObservedRunningTime="2026-07-17 21:34:53.234299811 +0000 UTC m=+430.504297514" Jul 17 21:35:04.220387 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:04.220309 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:35:20.605133 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.605097 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp"] Jul 17 21:35:20.605571 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.605487 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219b3cd0-0636-4ecd-85a7-06976b5b2900" containerName="console" Jul 17 21:35:20.605571 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.605499 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="219b3cd0-0636-4ecd-85a7-06976b5b2900" containerName="console" Jul 17 21:35:20.605571 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.605572 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="219b3cd0-0636-4ecd-85a7-06976b5b2900" containerName="console" Jul 17 21:35:20.613191 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.613167 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.615677 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.615655 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rc2tj\"" Jul 17 21:35:20.619332 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.619302 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp"] Jul 17 21:35:20.700000 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.699965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700181 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700181 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700056 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700319 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700177 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700319 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700405 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700318 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700405 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfq5\" (UniqueName: \"kubernetes.io/projected/6eb8db5f-eacd-4224-9371-01f89f6ad86c-kube-api-access-jhfq5\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700484 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700411 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.700484 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.700455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801318 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801278 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801318 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801318 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801519 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801554 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfq5\" (UniqueName: \"kubernetes.io/projected/6eb8db5f-eacd-4224-9371-01f89f6ad86c-kube-api-access-jhfq5\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801590 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801564 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801641 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801690 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801742 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801700 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801742 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801848 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801787 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.801848 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.801830 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.802077 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.802030 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.802219 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.802197 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.802322 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.802297 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.803709 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.803691 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.804042 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.804023 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.809413 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.809388 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eb8db5f-eacd-4224-9371-01f89f6ad86c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.809569 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.809552 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfq5\" (UniqueName: \"kubernetes.io/projected/6eb8db5f-eacd-4224-9371-01f89f6ad86c-kube-api-access-jhfq5\") pod \"maas-default-gateway-openshift-default-5896d4dd49-zzvnp\" (UID: \"6eb8db5f-eacd-4224-9371-01f89f6ad86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:20.924297 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:20.924270 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:21.059258 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.059127 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp"] Jul 17 21:35:21.061760 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:35:21.061726 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb8db5f_eacd_4224_9371_01f89f6ad86c.slice/crio-4a77ff354a6e2e3f1a5ce8db0a09db1e50014d7539b78ee05d6c9834ff639a57 WatchSource:0}: Error finding container 4a77ff354a6e2e3f1a5ce8db0a09db1e50014d7539b78ee05d6c9834ff639a57: Status 404 returned error can't find the container with id 4a77ff354a6e2e3f1a5ce8db0a09db1e50014d7539b78ee05d6c9834ff639a57 Jul 17 21:35:21.064195 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.064149 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:35:21.064295 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.064284 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:35:21.064338 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.064321 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:35:21.343514 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.343418 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" event={"ID":"6eb8db5f-eacd-4224-9371-01f89f6ad86c","Type":"ContainerStarted","Data":"49dcaae94ec6045f440caccbc0a7bde650c7adc4889fcf685d2c64bdd2510626"} Jul 17 21:35:21.343514 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.343485 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" event={"ID":"6eb8db5f-eacd-4224-9371-01f89f6ad86c","Type":"ContainerStarted","Data":"4a77ff354a6e2e3f1a5ce8db0a09db1e50014d7539b78ee05d6c9834ff639a57"} Jul 17 21:35:21.364641 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.364591 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" podStartSLOduration=1.364573046 podStartE2EDuration="1.364573046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:21.36261271 +0000 UTC m=+458.632610448" watchObservedRunningTime="2026-07-17 21:35:21.364573046 +0000 UTC m=+458.634570750" Jul 17 21:35:21.925011 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:21.924977 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:22.686327 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.686290 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:35:22.689939 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.689917 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:22.692483 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.692460 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 21:35:22.692591 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.692541 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kf4g2\"" Jul 17 21:35:22.699090 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.699065 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:35:22.716999 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.716958 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:35:22.818876 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.818839 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7dt\" (UniqueName: \"kubernetes.io/projected/4081ac11-e146-4bbb-b6d0-aee0f5826429-kube-api-access-9h7dt\") pod \"limitador-limitador-54d6d988d5-jxrtm\" (UID: \"4081ac11-e146-4bbb-b6d0-aee0f5826429\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:22.818876 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.818881 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4081ac11-e146-4bbb-b6d0-aee0f5826429-config-file\") pod \"limitador-limitador-54d6d988d5-jxrtm\" (UID: \"4081ac11-e146-4bbb-b6d0-aee0f5826429\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:22.919406 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.919366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7dt\" (UniqueName: \"kubernetes.io/projected/4081ac11-e146-4bbb-b6d0-aee0f5826429-kube-api-access-9h7dt\") pod \"limitador-limitador-54d6d988d5-jxrtm\" (UID: \"4081ac11-e146-4bbb-b6d0-aee0f5826429\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:22.919593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.919419 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4081ac11-e146-4bbb-b6d0-aee0f5826429-config-file\") pod \"limitador-limitador-54d6d988d5-jxrtm\" (UID: \"4081ac11-e146-4bbb-b6d0-aee0f5826429\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:22.920056 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.920037 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4081ac11-e146-4bbb-b6d0-aee0f5826429-config-file\") pod \"limitador-limitador-54d6d988d5-jxrtm\" (UID: \"4081ac11-e146-4bbb-b6d0-aee0f5826429\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:22.928020 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.927995 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7dt\" (UniqueName: \"kubernetes.io/projected/4081ac11-e146-4bbb-b6d0-aee0f5826429-kube-api-access-9h7dt\") pod \"limitador-limitador-54d6d988d5-jxrtm\" (UID: \"4081ac11-e146-4bbb-b6d0-aee0f5826429\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:22.931278 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:22.931256 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:23.001833 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:23.001728 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:23.142021 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:23.141998 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:35:23.144762 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:35:23.144733 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4081ac11_e146_4bbb_b6d0_aee0f5826429.slice/crio-dc99bae748cfab1cf77baf506e94e9818506826f4ee9643944672e5dea55b394 WatchSource:0}: Error finding container dc99bae748cfab1cf77baf506e94e9818506826f4ee9643944672e5dea55b394: Status 404 returned error can't find the container with id dc99bae748cfab1cf77baf506e94e9818506826f4ee9643944672e5dea55b394 Jul 17 21:35:23.354844 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:23.354752 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" event={"ID":"4081ac11-e146-4bbb-b6d0-aee0f5826429","Type":"ContainerStarted","Data":"dc99bae748cfab1cf77baf506e94e9818506826f4ee9643944672e5dea55b394"} Jul 17 21:35:23.355013 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:23.354954 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:23.356046 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:23.356026 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:35:26.372667 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:26.372571 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" event={"ID":"4081ac11-e146-4bbb-b6d0-aee0f5826429","Type":"ContainerStarted","Data":"5630fefa4c6a80de425592c6bbad052cb937875e46a6124cadd15fd68bfb0a75"} Jul 17 21:35:26.373065 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:26.372716 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:35:26.388940 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:26.388890 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" podStartSLOduration=1.5209501429999999 podStartE2EDuration="4.388873704s" totalImagesPullingTime="2.867923561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:35:26.388151784 +0000 UTC m=+463.658149508" watchObservedRunningTime="2026-07-17 21:35:26.388873704 +0000 UTC m=+463.658871409" Jul 17 21:35:37.378156 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:35:37.378125 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jxrtm" Jul 17 21:36:32.695765 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:32.695727 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:36:39.396456 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:39.396416 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-zzvnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 21:36:39.396844 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:39.396499 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" podUID="6eb8db5f-eacd-4224-9371-01f89f6ad86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 21:36:40.721890 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:40.721856 2566 generic.go:356] "Generic (PLEG): container finished" podID="6eb8db5f-eacd-4224-9371-01f89f6ad86c" containerID="49dcaae94ec6045f440caccbc0a7bde650c7adc4889fcf685d2c64bdd2510626" exitCode=0 Jul 17 21:36:40.722328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:40.721933 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" event={"ID":"6eb8db5f-eacd-4224-9371-01f89f6ad86c","Type":"ContainerDied","Data":"49dcaae94ec6045f440caccbc0a7bde650c7adc4889fcf685d2c64bdd2510626"} Jul 17 21:36:40.722373 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:40.722334 2566 scope.go:117] "RemoveContainer" containerID="49dcaae94ec6045f440caccbc0a7bde650c7adc4889fcf685d2c64bdd2510626" Jul 17 21:36:40.723358 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:40.723332 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:36:40.723420 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:40.723401 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:36:40.723454 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:40.723435 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:36:41.728703 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:41.728673 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" event={"ID":"6eb8db5f-eacd-4224-9371-01f89f6ad86c","Type":"ContainerStarted","Data":"228f07d44104790af79d65d0f12875f09fb2593517580039e31b234b9c3717bf"} Jul 17 21:36:41.925169 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:41.925136 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:36:41.925505 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:41.925485 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-zzvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 17 21:36:41.925556 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:41.925536 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" podUID="6eb8db5f-eacd-4224-9371-01f89f6ad86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 17 21:36:42.924977 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:42.924942 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-zzvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 17 21:36:42.925446 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:42.924998 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" podUID="6eb8db5f-eacd-4224-9371-01f89f6ad86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 17 21:36:43.924811 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:43.924776 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-zzvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 17 21:36:43.924992 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:43.924833 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" podUID="6eb8db5f-eacd-4224-9371-01f89f6ad86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 17 21:36:44.925405 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:44.925370 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-zzvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 17 21:36:44.925879 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:44.925449 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" podUID="6eb8db5f-eacd-4224-9371-01f89f6ad86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 17 21:36:45.928629 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:45.928599 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:36:45.929005 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:45.928852 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:36:45.929534 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:45.929516 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-zzvnp" Jul 17 21:36:47.903750 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:47.903716 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:36:53.299400 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:36:53.299371 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:37:11.312358 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:11.312329 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:37:46.141363 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.141324 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7"] Jul 17 21:37:46.144873 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.144852 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.145063 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.145039 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:37:46.150162 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.149972 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ghszk\"" Jul 17 21:37:46.150162 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.150028 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 21:37:46.150162 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.150069 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 21:37:46.150162 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.150028 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 17 21:37:46.160831 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.160808 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7"] Jul 17 21:37:46.217725 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.217695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.217906 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.217733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2870c21-fe65-405c-a686-238eee98e265-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.217906 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.217807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.218032 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.217941 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dl7\" (UniqueName: \"kubernetes.io/projected/a2870c21-fe65-405c-a686-238eee98e265-kube-api-access-p7dl7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.218032 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.217970 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.218150 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.218042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.318979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319026 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2870c21-fe65-405c-a686-238eee98e265-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319280 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319280 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319250 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dl7\" (UniqueName: \"kubernetes.io/projected/a2870c21-fe65-405c-a686-238eee98e265-kube-api-access-p7dl7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319355 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319307 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319355 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319451 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319543 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319523 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.319610 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.319594 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.321562 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.321533 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2870c21-fe65-405c-a686-238eee98e265-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.321767 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.321751 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2870c21-fe65-405c-a686-238eee98e265-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.332773 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.332739 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dl7\" (UniqueName: \"kubernetes.io/projected/a2870c21-fe65-405c-a686-238eee98e265-kube-api-access-p7dl7\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ht9s7\" (UID: \"a2870c21-fe65-405c-a686-238eee98e265\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.456005 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.455976 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:46.608735 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:46.608707 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7"] Jul 17 21:37:46.611543 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:37:46.611515 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2870c21_fe65_405c_a686_238eee98e265.slice/crio-e696836e65f8a425625173b134e38acacc5c1510e9729f81afe5e56165f2bc35 WatchSource:0}: Error finding container e696836e65f8a425625173b134e38acacc5c1510e9729f81afe5e56165f2bc35: Status 404 returned error can't find the container with id e696836e65f8a425625173b134e38acacc5c1510e9729f81afe5e56165f2bc35 Jul 17 21:37:47.036971 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:47.036931 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" event={"ID":"a2870c21-fe65-405c-a686-238eee98e265","Type":"ContainerStarted","Data":"e696836e65f8a425625173b134e38acacc5c1510e9729f81afe5e56165f2bc35"} Jul 17 21:37:48.043172 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:48.043131 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" event={"ID":"a2870c21-fe65-405c-a686-238eee98e265","Type":"ContainerStarted","Data":"05c4cf3baf2d2c17af40ebaf6637297210dfa8fed1cf35e33cf04852dd270f1d"} Jul 17 21:37:48.062389 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:48.062339 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" podStartSLOduration=1.033489413 podStartE2EDuration="2.06232128s" totalImagesPullingTime="1.028831867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:37:48.061776518 +0000 UTC m=+605.331774223" watchObservedRunningTime="2026-07-17 21:37:48.06232128 +0000 UTC m=+605.332318962" Jul 17 21:37:54.228991 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:54.228953 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:37:56.456886 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:56.456852 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:56.457321 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:56.456978 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:56.469537 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:56.469513 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:37:57.098990 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:37:57.098961 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ht9s7" Jul 17 21:39:30.117395 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:39:30.117351 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:39:41.707192 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:39:41.707156 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:39:50.919248 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:39:50.919198 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:41:30.021284 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:41:30.021188 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:41:44.305200 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:41:44.305159 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:42:19.605345 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:42:19.605305 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:42:29.011116 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:42:29.011076 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:42:39.715546 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:42:39.715511 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:42:48.004392 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:42:48.004353 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:42:59.415648 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:42:59.415614 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:43:11.219086 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:43:11.219044 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:43:21.602694 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:43:21.602657 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:43:31.403954 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:43:31.403914 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:43:57.013802 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:43:57.013762 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:44:09.607662 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:44:09.607580 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:44:32.011849 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:44:32.011811 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:44:48.206703 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:44:48.206667 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:44:58.310743 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:44:58.310706 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:45:00.144536 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:00.144498 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb"] Jul 17 21:45:00.148147 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:00.148128 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" Jul 17 21:45:00.151751 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:00.151725 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 21:45:00.158162 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:00.158136 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" Jul 17 21:45:00.169645 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:00.169619 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb"] Jul 17 21:45:00.297431 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:00.297408 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb"] Jul 17 21:45:00.299615 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:45:00.299585 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bab2e9d_a935_4bca_a95f_f47662745737.slice/crio-066febf76e11e42d1251c11aa60494259d3693ec8c0637d7069452647a6b29a4 WatchSource:0}: Error finding container 066febf76e11e42d1251c11aa60494259d3693ec8c0637d7069452647a6b29a4: Status 404 returned error can't find the container with id 066febf76e11e42d1251c11aa60494259d3693ec8c0637d7069452647a6b29a4 Jul 17 21:45:00.301779 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:00.301760 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:45:01.075175 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:01.075141 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" event={"ID":"0bab2e9d-a935-4bca-a95f-f47662745737","Type":"ContainerStarted","Data":"066febf76e11e42d1251c11aa60494259d3693ec8c0637d7069452647a6b29a4"} Jul 17 21:45:03.087611 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:03.087576 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" event={"ID":"0bab2e9d-a935-4bca-a95f-f47662745737","Type":"ContainerStarted","Data":"67fd82f9693eb7e53337399fee909654054445cdbdf369d6bebcaf8b24ac2a27"} Jul 17 21:45:03.104497 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:03.104445 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" podStartSLOduration=0.760324959 podStartE2EDuration="3.104428958s" totalImagesPullingTime="2.344103999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:45:03.102125512 +0000 UTC m=+1040.372123227" watchObservedRunningTime="2026-07-17 21:45:03.104428958 +0000 UTC m=+1040.374426731" Jul 17 21:45:04.093770 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:04.093731 2566 generic.go:356] "Generic (PLEG): container finished" podID="0bab2e9d-a935-4bca-a95f-f47662745737" containerID="67fd82f9693eb7e53337399fee909654054445cdbdf369d6bebcaf8b24ac2a27" exitCode=0 Jul 17 21:45:04.094141 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:04.093783 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" event={"ID":"0bab2e9d-a935-4bca-a95f-f47662745737","Type":"ContainerDied","Data":"67fd82f9693eb7e53337399fee909654054445cdbdf369d6bebcaf8b24ac2a27"} Jul 17 21:45:05.230791 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:05.230761 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" Jul 17 21:45:06.104892 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:06.104850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" event={"ID":"0bab2e9d-a935-4bca-a95f-f47662745737","Type":"ContainerDied","Data":"066febf76e11e42d1251c11aa60494259d3693ec8c0637d7069452647a6b29a4"} Jul 17 21:45:06.105070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:06.104912 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066febf76e11e42d1251c11aa60494259d3693ec8c0637d7069452647a6b29a4" Jul 17 21:45:06.105070 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:06.104864 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-5xhxb" Jul 17 21:45:15.613301 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:15.613265 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:45:26.103958 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:26.103920 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:45:35.813220 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:45:35.813185 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:46:03.150105 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:03.150064 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc"] Jul 17 21:46:03.150526 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:03.150283 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" podUID="4f27bbac-968e-4765-9080-05b85d408e25" containerName="manager" containerID="cri-o://f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f" gracePeriod=10 Jul 17 21:46:05.608869 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:05.608841 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:46:05.647741 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:05.647662 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwgxl\" (UniqueName: \"kubernetes.io/projected/4f27bbac-968e-4765-9080-05b85d408e25-kube-api-access-vwgxl\") pod \"4f27bbac-968e-4765-9080-05b85d408e25\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " Jul 17 21:46:05.647741 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:05.647738 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f27bbac-968e-4765-9080-05b85d408e25-extensions-socket-volume\") pod \"4f27bbac-968e-4765-9080-05b85d408e25\" (UID: \"4f27bbac-968e-4765-9080-05b85d408e25\") " Jul 17 21:46:05.648132 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:05.648103 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27bbac-968e-4765-9080-05b85d408e25-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4f27bbac-968e-4765-9080-05b85d408e25" (UID: "4f27bbac-968e-4765-9080-05b85d408e25"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:46:05.649934 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:05.649909 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f27bbac-968e-4765-9080-05b85d408e25-kube-api-access-vwgxl" (OuterVolumeSpecName: "kube-api-access-vwgxl") pod "4f27bbac-968e-4765-9080-05b85d408e25" (UID: "4f27bbac-968e-4765-9080-05b85d408e25"). InnerVolumeSpecName "kube-api-access-vwgxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:46:05.749156 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:05.749121 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwgxl\" (UniqueName: \"kubernetes.io/projected/4f27bbac-968e-4765-9080-05b85d408e25-kube-api-access-vwgxl\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:46:05.749156 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:05.749149 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f27bbac-968e-4765-9080-05b85d408e25-extensions-socket-volume\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:46:06.386102 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.386069 2566 generic.go:356] "Generic (PLEG): container finished" podID="4f27bbac-968e-4765-9080-05b85d408e25" containerID="f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f" exitCode=0 Jul 17 21:46:06.386300 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.386119 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" Jul 17 21:46:06.386300 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.386152 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" event={"ID":"4f27bbac-968e-4765-9080-05b85d408e25","Type":"ContainerDied","Data":"f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f"} Jul 17 21:46:06.386300 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.386203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc" event={"ID":"4f27bbac-968e-4765-9080-05b85d408e25","Type":"ContainerDied","Data":"3fccc00d2075986b682a4ed7592daadebef0cee3bb08aead57ce44a101298c04"} Jul 17 21:46:06.386300 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.386239 2566 scope.go:117] "RemoveContainer" containerID="f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f" Jul 17 21:46:06.397153 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.397136 2566 scope.go:117] "RemoveContainer" containerID="f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f" Jul 17 21:46:06.397460 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:46:06.397439 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f\": container with ID starting with f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f not found: ID does not exist" containerID="f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f" Jul 17 21:46:06.397541 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.397468 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f"} err="failed to get container status \"f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f\": rpc error: code = NotFound desc = could not find container \"f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f\": container with ID starting with f6dbb4b2f8df56e910d29d866f24a48bb7e12db846aebd01841780f42fd6693f not found: ID does not exist" Jul 17 21:46:06.408580 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.408554 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc"] Jul 17 21:46:06.412399 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:06.412378 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9hcbc"] Jul 17 21:46:07.299089 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:46:07.299054 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f27bbac-968e-4765-9080-05b85d408e25" path="/var/lib/kubelet/pods/4f27bbac-968e-4765-9080-05b85d408e25/volumes" Jul 17 21:47:04.313517 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.313440 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77"] Jul 17 21:47:04.315803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.313829 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bab2e9d-a935-4bca-a95f-f47662745737" containerName="cleanup" Jul 17 21:47:04.315803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.313842 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bab2e9d-a935-4bca-a95f-f47662745737" containerName="cleanup" Jul 17 21:47:04.315803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.313856 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f27bbac-968e-4765-9080-05b85d408e25" containerName="manager" Jul 17 21:47:04.315803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.313863 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f27bbac-968e-4765-9080-05b85d408e25" containerName="manager" Jul 17 21:47:04.315803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.313917 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f27bbac-968e-4765-9080-05b85d408e25" containerName="manager" Jul 17 21:47:04.315803 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.313928 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bab2e9d-a935-4bca-a95f-f47662745737" containerName="cleanup" Jul 17 21:47:04.316776 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.316759 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.319280 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.319262 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhf9w\"" Jul 17 21:47:04.332618 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.332592 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77"] Jul 17 21:47:04.373677 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.373642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece64894-51b5-488d-8fb3-7864be73d0d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5kz77\" (UID: \"ece64894-51b5-488d-8fb3-7864be73d0d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.373874 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.373712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspx5\" (UniqueName: \"kubernetes.io/projected/ece64894-51b5-488d-8fb3-7864be73d0d0-kube-api-access-nspx5\") pod \"kuadrant-operator-controller-manager-86878c769f-5kz77\" (UID: \"ece64894-51b5-488d-8fb3-7864be73d0d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.474460 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.474421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece64894-51b5-488d-8fb3-7864be73d0d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5kz77\" (UID: \"ece64894-51b5-488d-8fb3-7864be73d0d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.474460 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.474466 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspx5\" (UniqueName: \"kubernetes.io/projected/ece64894-51b5-488d-8fb3-7864be73d0d0-kube-api-access-nspx5\") pod \"kuadrant-operator-controller-manager-86878c769f-5kz77\" (UID: \"ece64894-51b5-488d-8fb3-7864be73d0d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.474862 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.474837 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece64894-51b5-488d-8fb3-7864be73d0d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5kz77\" (UID: \"ece64894-51b5-488d-8fb3-7864be73d0d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.482866 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.482842 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspx5\" (UniqueName: \"kubernetes.io/projected/ece64894-51b5-488d-8fb3-7864be73d0d0-kube-api-access-nspx5\") pod \"kuadrant-operator-controller-manager-86878c769f-5kz77\" (UID: \"ece64894-51b5-488d-8fb3-7864be73d0d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.627000 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.626914 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:04.761838 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:04.761803 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77"] Jul 17 21:47:04.765039 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:47:04.765013 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece64894_51b5_488d_8fb3_7864be73d0d0.slice/crio-9b727ce8f034d76b8587e08a2caf8a76a6ea3e8dd96abfde6c7db98a1048d4b1 WatchSource:0}: Error finding container 9b727ce8f034d76b8587e08a2caf8a76a6ea3e8dd96abfde6c7db98a1048d4b1: Status 404 returned error can't find the container with id 9b727ce8f034d76b8587e08a2caf8a76a6ea3e8dd96abfde6c7db98a1048d4b1 Jul 17 21:47:05.656640 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:05.656603 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" event={"ID":"ece64894-51b5-488d-8fb3-7864be73d0d0","Type":"ContainerStarted","Data":"c4c48d96ebbd7ee2269c27b05aa564212912883cfaa4086b49998420515693da"} Jul 17 21:47:05.657045 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:05.656658 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" event={"ID":"ece64894-51b5-488d-8fb3-7864be73d0d0","Type":"ContainerStarted","Data":"9b727ce8f034d76b8587e08a2caf8a76a6ea3e8dd96abfde6c7db98a1048d4b1"} Jul 17 21:47:05.657045 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:05.656741 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:05.679537 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:05.679487 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" podStartSLOduration=1.679468679 podStartE2EDuration="1.679468679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:05.676852457 +0000 UTC m=+1162.946850166" watchObservedRunningTime="2026-07-17 21:47:05.679468679 +0000 UTC m=+1162.949466438" Jul 17 21:47:16.663470 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:16.663437 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5kz77" Jul 17 21:47:24.912372 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:24.912334 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:47:38.530830 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:38.530791 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:47:56.505293 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:47:56.505258 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:48:06.009680 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:48:06.009647 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:48:17.003790 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:48:17.003759 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:48:25.605013 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:48:25.604977 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:48:36.606183 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:48:36.606144 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:48:47.802301 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:48:47.802263 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:48:56.507593 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:48:56.507553 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:49:07.605151 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:49:07.605115 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:49:29.006973 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:49:29.006936 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:49:38.004071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:49:38.004031 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:49:48.301377 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:49:48.301341 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:49:59.805548 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:49:59.805515 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:50:07.898848 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:50:07.898764 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:50:19.306857 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:50:19.306822 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:50:28.406280 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:50:28.406244 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:50:37.608964 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:50:37.608931 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:50:45.612983 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:50:45.612945 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:50:57.012266 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:50:57.012222 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:51:06.411435 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:51:06.411393 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:51:18.806410 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:51:18.806376 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:51:27.610459 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:51:27.610424 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:51:37.011749 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:51:37.011714 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:51:47.614674 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:51:47.614639 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:52:01.005544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:52:01.005512 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:52:07.700598 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:52:07.700559 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:52:24.512284 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:52:24.512243 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:52:33.902515 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:52:33.902479 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:52:43.603461 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:52:43.603425 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:52:50.306643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:52:50.306605 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:53:03.810096 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:03.810015 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:53:13.702402 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:13.702363 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:53:22.925142 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.925107 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn"] Jul 17 21:53:22.928218 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.928195 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.930741 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.930716 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f9bc6369-openshift-default-dockercfg-jpcxw\"" Jul 17 21:53:22.952212 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.952179 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn"] Jul 17 21:53:22.984382 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984351 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-podinfo\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984589 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984390 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-certs\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984589 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984419 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-envoy\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984589 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984435 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-data\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984589 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgxw\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-kube-api-access-ntgxw\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984896 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-credential-socket\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984896 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984640 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-token\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984896 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984678 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-socket\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:22.984896 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:22.984732 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istiod-ca-cert\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086100 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086067 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istiod-ca-cert\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086285 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-podinfo\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086285 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086148 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-certs\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086285 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086179 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-envoy\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086285 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-data\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086516 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgxw\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-kube-api-access-ntgxw\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086516 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-credential-socket\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086516 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086367 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-token\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086516 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086401 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-socket\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086761 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086657 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-certs\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086761 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-data\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.086953 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086920 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-credential-socket\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.087066 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.086899 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istiod-ca-cert\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.087114 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.087087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-socket\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.088671 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.088651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-envoy\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.088898 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.088880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-podinfo\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.096812 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.096789 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-token\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.097002 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.096984 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgxw\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-kube-api-access-ntgxw\") pod \"e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.239341 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.239220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:23.380988 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.380963 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn"] Jul 17 21:53:23.383051 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:53:23.383027 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9233fa_cdc4_4ad2_9e5f_03fc1f4ee858.slice/crio-9cf6fbb7736cf88933fee6289ba7b294a1f5125541633ca1bcc9a7394bc7eeb2 WatchSource:0}: Error finding container 9cf6fbb7736cf88933fee6289ba7b294a1f5125541633ca1bcc9a7394bc7eeb2: Status 404 returned error can't find the container with id 9cf6fbb7736cf88933fee6289ba7b294a1f5125541633ca1bcc9a7394bc7eeb2 Jul 17 21:53:23.385168 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.385136 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:53:23.385261 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.385199 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:53:23.385297 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.385257 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:53:23.446027 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:23.445999 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" event={"ID":"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858","Type":"ContainerStarted","Data":"9cf6fbb7736cf88933fee6289ba7b294a1f5125541633ca1bcc9a7394bc7eeb2"} Jul 17 21:53:24.453568 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:24.453531 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" event={"ID":"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858","Type":"ContainerStarted","Data":"edc10ee5c97b53f7167e915e6906833bd9c70aa58cc00ab1e4a53ea8bc25b049"} Jul 17 21:53:24.473521 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:24.473473 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" podStartSLOduration=2.4734559369999998 podStartE2EDuration="2.473455937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:24.471925055 +0000 UTC m=+1541.741922781" watchObservedRunningTime="2026-07-17 21:53:24.473455937 +0000 UTC m=+1541.743453641" Jul 17 21:53:25.143853 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:25.143817 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:53:25.239806 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:25.239772 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:25.244576 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:25.244550 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:25.458151 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:25.458121 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:25.459323 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:25.459302 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:36.807291 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:36.807250 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:53:39.524076 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:39.524040 2566 generic.go:356] "Generic (PLEG): container finished" podID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerID="edc10ee5c97b53f7167e915e6906833bd9c70aa58cc00ab1e4a53ea8bc25b049" exitCode=0 Jul 17 21:53:39.524535 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:39.524110 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" event={"ID":"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858","Type":"ContainerDied","Data":"edc10ee5c97b53f7167e915e6906833bd9c70aa58cc00ab1e4a53ea8bc25b049"} Jul 17 21:53:39.524535 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:39.524529 2566 scope.go:117] "RemoveContainer" containerID="edc10ee5c97b53f7167e915e6906833bd9c70aa58cc00ab1e4a53ea8bc25b049" Jul 17 21:53:39.525643 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:39.525615 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:53:39.525711 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:39.525681 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:53:39.525748 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:39.525715 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:53:40.458850 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:40.458739 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:40.530508 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:40.530475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" event={"ID":"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858","Type":"ContainerStarted","Data":"5d3416e230bc749aa804c596bb18478d8a6dec1d4db763aa07b912cce50d9e20"} Jul 17 21:53:41.239612 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:41.239576 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:41.240742 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:41.240716 2566 patch_prober.go:28] interesting pod/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 17 21:53:41.240851 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:41.240763 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 17 21:53:42.239850 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:42.239814 2566 patch_prober.go:28] interesting pod/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 17 21:53:42.240295 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:42.239873 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 17 21:53:43.240419 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:43.240384 2566 patch_prober.go:28] interesting pod/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 17 21:53:43.240814 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:43.240439 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 17 21:53:44.239954 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:44.239912 2566 patch_prober.go:28] interesting pod/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 17 21:53:44.240170 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:44.239976 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 17 21:53:45.244411 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:45.244378 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:45.244861 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:45.244665 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:53:45.245332 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:53:45.245313 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:56:29.242777 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:29.242686 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn"] Jul 17 21:56:29.243382 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:29.242955 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" containerID="cri-o://5d3416e230bc749aa804c596bb18478d8a6dec1d4db763aa07b912cce50d9e20" gracePeriod=30 Jul 17 21:56:30.247667 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:30.247629 2566 patch_prober.go:28] interesting pod/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 21:56:30.248029 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:30.247701 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 21:56:32.002941 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:32.002905 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:56:34.371879 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.371844 2566 generic.go:356] "Generic (PLEG): container finished" podID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerID="5d3416e230bc749aa804c596bb18478d8a6dec1d4db763aa07b912cce50d9e20" exitCode=0 Jul 17 21:56:34.372323 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.371913 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" event={"ID":"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858","Type":"ContainerDied","Data":"5d3416e230bc749aa804c596bb18478d8a6dec1d4db763aa07b912cce50d9e20"} Jul 17 21:56:34.372323 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.371979 2566 scope.go:117] "RemoveContainer" containerID="edc10ee5c97b53f7167e915e6906833bd9c70aa58cc00ab1e4a53ea8bc25b049" Jul 17 21:56:34.502291 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.502268 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:56:34.610531 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610499 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-podinfo\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.610531 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610537 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-data\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.610787 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610562 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntgxw\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-kube-api-access-ntgxw\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.610787 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610590 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-envoy\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.610787 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610656 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-credential-socket\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.610787 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610687 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istiod-ca-cert\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.610787 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610765 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-token\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.611042 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.610795 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-certs\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.611097 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611047 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:56:34.611149 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611110 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:56:34.611208 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611192 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-socket\") pod \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\" (UID: \"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858\") " Jul 17 21:56:34.611281 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611201 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:56:34.611421 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611398 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:56:34.611608 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611581 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:56:34.611793 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611775 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.611865 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611798 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.611865 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611812 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.611865 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611824 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.611865 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.611839 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.613160 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.613135 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-kube-api-access-ntgxw" (OuterVolumeSpecName: "kube-api-access-ntgxw") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "kube-api-access-ntgxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:56:34.613160 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.613152 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 21:56:34.613309 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.613161 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:56:34.613411 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.613391 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" (UID: "3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:56:34.712994 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.712958 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntgxw\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-kube-api-access-ntgxw\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.712994 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.712986 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.712994 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.712995 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:34.712994 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:34.713003 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:56:35.378328 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:35.378300 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" Jul 17 21:56:35.378742 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:35.378329 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn" event={"ID":"3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858","Type":"ContainerDied","Data":"9cf6fbb7736cf88933fee6289ba7b294a1f5125541633ca1bcc9a7394bc7eeb2"} Jul 17 21:56:35.378742 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:35.378383 2566 scope.go:117] "RemoveContainer" containerID="5d3416e230bc749aa804c596bb18478d8a6dec1d4db763aa07b912cce50d9e20" Jul 17 21:56:35.396739 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:35.396713 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn"] Jul 17 21:56:35.401867 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:35.401844 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f9bc6369-openshift-default-7bb955c44c-pkcsn"] Jul 17 21:56:37.298136 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:37.298098 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" path="/var/lib/kubelet/pods/3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858/volumes" Jul 17 21:56:41.242966 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.242923 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg"] Jul 17 21:56:41.243975 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.243943 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" Jul 17 21:56:41.243975 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.243973 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" Jul 17 21:56:41.244119 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.243995 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" Jul 17 21:56:41.244119 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.244004 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" Jul 17 21:56:41.244119 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.244098 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" Jul 17 21:56:41.244119 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.244111 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9233fa-cdc4-4ad2-9e5f-03fc1f4ee858" containerName="istio-proxy" Jul 17 21:56:41.249144 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.249066 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.251966 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.251943 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-281d4ac2-gw-openshift-default-dockercfg-2bfft\"" Jul 17 21:56:41.261785 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.261753 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg"] Jul 17 21:56:41.375560 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375528 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-envoy\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.375781 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375583 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-socket\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.375781 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375619 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-certs\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.375781 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvjj\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-kube-api-access-nxvjj\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.375781 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375740 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-istio-token\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.375781 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-data\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.376019 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c75f93-c142-469e-a289-34743fabf496-istio-podinfo\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.376019 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c75f93-c142-469e-a289-34743fabf496-istiod-ca-cert\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.376019 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.375866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-credential-socket\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477045 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-envoy\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477224 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477065 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-socket\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477224 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477094 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-certs\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477224 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477114 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvjj\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-kube-api-access-nxvjj\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477224 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477136 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-istio-token\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477224 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-data\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477224 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c75f93-c142-469e-a289-34743fabf496-istio-podinfo\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477224 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c75f93-c142-469e-a289-34743fabf496-istiod-ca-cert\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477631 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477574 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-socket\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477631 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477619 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-data\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477722 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-credential-socket\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477722 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477682 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-certs\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.477983 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.477965 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c75f93-c142-469e-a289-34743fabf496-istiod-ca-cert\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.478090 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.478071 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-credential-socket\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.479664 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.479635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-envoy\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.480078 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.480060 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c75f93-c142-469e-a289-34743fabf496-istio-podinfo\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.485485 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.485460 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvjj\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-kube-api-access-nxvjj\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.485655 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.485635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-istio-token\") pod \"e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.563549 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.563451 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:41.711843 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.711815 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg"] Jul 17 21:56:41.715076 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:56:41.715045 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c75f93_c142_469e_a289_34743fabf496.slice/crio-63eab3c3d94ed13606f4ef6a3a4db70789873d29b5e8923d6ef10b38ce9db8af WatchSource:0}: Error finding container 63eab3c3d94ed13606f4ef6a3a4db70789873d29b5e8923d6ef10b38ce9db8af: Status 404 returned error can't find the container with id 63eab3c3d94ed13606f4ef6a3a4db70789873d29b5e8923d6ef10b38ce9db8af Jul 17 21:56:41.717178 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.717146 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:56:41.717252 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.717216 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:56:41.717301 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:41.717266 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:56:42.413198 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:42.413156 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" event={"ID":"92c75f93-c142-469e-a289-34743fabf496","Type":"ContainerStarted","Data":"7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d"} Jul 17 21:56:42.413697 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:42.413216 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" event={"ID":"92c75f93-c142-469e-a289-34743fabf496","Type":"ContainerStarted","Data":"63eab3c3d94ed13606f4ef6a3a4db70789873d29b5e8923d6ef10b38ce9db8af"} Jul 17 21:56:42.435838 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:42.435784 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" podStartSLOduration=1.435766543 podStartE2EDuration="1.435766543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:42.433168364 +0000 UTC m=+1739.703166090" watchObservedRunningTime="2026-07-17 21:56:42.435766543 +0000 UTC m=+1739.705764250" Jul 17 21:56:42.563752 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:42.563691 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:42.568677 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:42.568645 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:43.418505 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:43.418468 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:43.419597 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:43.419573 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:45.412578 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:45.412541 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:56:53.472498 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:53.472463 2566 generic.go:356] "Generic (PLEG): container finished" podID="92c75f93-c142-469e-a289-34743fabf496" containerID="7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d" exitCode=0 Jul 17 21:56:53.473006 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:53.472541 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" event={"ID":"92c75f93-c142-469e-a289-34743fabf496","Type":"ContainerDied","Data":"7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d"} Jul 17 21:56:53.473071 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:53.473048 2566 scope.go:117] "RemoveContainer" containerID="7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d" Jul 17 21:56:53.474109 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:53.474079 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:56:53.474206 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:53.474147 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:56:53.474206 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:53.474178 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:56:54.480093 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:54.480059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" event={"ID":"92c75f93-c142-469e-a289-34743fabf496","Type":"ContainerStarted","Data":"35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2"} Jul 17 21:56:54.564011 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:54.563917 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:54.564716 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:54.564680 2566 patch_prober.go:28] interesting pod/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 21:56:54.564833 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:54.564757 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 21:56:55.564888 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:55.564848 2566 patch_prober.go:28] interesting pod/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 21:56:55.565895 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:55.565471 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 21:56:56.574258 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:56.572644 2566 patch_prober.go:28] interesting pod/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 21:56:56.574258 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:56.572705 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 21:56:57.564934 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:57.564891 2566 patch_prober.go:28] interesting pod/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 21:56:57.565298 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:57.565267 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 21:56:58.418604 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:58.418561 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:58.567702 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:58.567671 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:56:58.568891 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:56:58.568866 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:57:47.146096 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:47.146047 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg"] Jul 17 21:57:47.146606 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:47.146423 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" containerID="cri-o://35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2" gracePeriod=30 Jul 17 21:57:50.204495 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:50.204458 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:57:52.395984 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.395959 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:57:52.436011 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.435980 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c75f93-c142-469e-a289-34743fabf496-istiod-ca-cert\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436200 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-socket\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436200 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436054 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-certs\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436200 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436077 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-istio-token\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436200 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436123 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-credential-socket\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436200 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436168 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-data\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436478 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436203 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-envoy\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436478 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436283 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c75f93-c142-469e-a289-34743fabf496-istio-podinfo\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436478 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436322 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxvjj\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-kube-api-access-nxvjj\") pod \"92c75f93-c142-469e-a289-34743fabf496\" (UID: \"92c75f93-c142-469e-a289-34743fabf496\") " Jul 17 21:57:52.436478 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436374 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:57:52.436478 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436392 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c75f93-c142-469e-a289-34743fabf496-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:57:52.436705 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436642 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.436705 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436663 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c75f93-c142-469e-a289-34743fabf496-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.436834 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436811 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:57:52.436932 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.436904 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:57:52.437283 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.437219 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-data" (OuterVolumeSpecName: "istio-data") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:57:52.438781 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.438752 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92c75f93-c142-469e-a289-34743fabf496-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 21:57:52.439010 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.438995 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:57:52.439171 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.439154 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-kube-api-access-nxvjj" (OuterVolumeSpecName: "kube-api-access-nxvjj") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "kube-api-access-nxvjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:57:52.439264 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.439169 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-istio-token" (OuterVolumeSpecName: "istio-token") pod "92c75f93-c142-469e-a289-34743fabf496" (UID: "92c75f93-c142-469e-a289-34743fabf496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:57:52.538065 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.538021 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.538065 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.538060 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.538372 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.538076 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.538372 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.538091 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.538372 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.538105 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c75f93-c142-469e-a289-34743fabf496-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.538372 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.538119 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c75f93-c142-469e-a289-34743fabf496-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.538372 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.538135 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxvjj\" (UniqueName: \"kubernetes.io/projected/92c75f93-c142-469e-a289-34743fabf496-kube-api-access-nxvjj\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 21:57:52.759550 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.759453 2566 generic.go:356] "Generic (PLEG): container finished" podID="92c75f93-c142-469e-a289-34743fabf496" containerID="35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2" exitCode=0 Jul 17 21:57:52.759550 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.759495 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" event={"ID":"92c75f93-c142-469e-a289-34743fabf496","Type":"ContainerDied","Data":"35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2"} Jul 17 21:57:52.759550 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.759521 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" Jul 17 21:57:52.759550 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.759534 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg" event={"ID":"92c75f93-c142-469e-a289-34743fabf496","Type":"ContainerDied","Data":"63eab3c3d94ed13606f4ef6a3a4db70789873d29b5e8923d6ef10b38ce9db8af"} Jul 17 21:57:52.759550 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.759555 2566 scope.go:117] "RemoveContainer" containerID="35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2" Jul 17 21:57:52.770868 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.770835 2566 scope.go:117] "RemoveContainer" containerID="7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d" Jul 17 21:57:52.779004 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.778985 2566 scope.go:117] "RemoveContainer" containerID="35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2" Jul 17 21:57:52.779277 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:57:52.779247 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2\": container with ID starting with 35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2 not found: ID does not exist" containerID="35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2" Jul 17 21:57:52.779391 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.779287 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2"} err="failed to get container status \"35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2\": rpc error: code = NotFound desc = could not find container \"35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2\": container with ID starting with 35795467fde23555ffbd76407b6d6e8b65ca3940594f99ac46c437eb4f9a80c2 not found: ID does not exist" Jul 17 21:57:52.779391 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.779307 2566 scope.go:117] "RemoveContainer" containerID="7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d" Jul 17 21:57:52.779563 ip-10-0-139-178 kubenswrapper[2566]: E0717 21:57:52.779541 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d\": container with ID starting with 7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d not found: ID does not exist" containerID="7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d" Jul 17 21:57:52.779600 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.779571 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d"} err="failed to get container status \"7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d\": rpc error: code = NotFound desc = could not find container \"7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d\": container with ID starting with 7ee40fe2f2807c1b01b1800310220f3d62d7c738b097cf5a2190d4301d4b934d not found: ID does not exist" Jul 17 21:57:52.784511 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.784487 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg"] Jul 17 21:57:52.785993 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:52.785974 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-281d4ac2-gw-openshift-default-7df5f9cb59-rsptg"] Jul 17 21:57:53.297901 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:53.297867 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c75f93-c142-469e-a289-34743fabf496" path="/var/lib/kubelet/pods/92c75f93-c142-469e-a289-34743fabf496/volumes" Jul 17 21:57:58.763596 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.763563 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf"] Jul 17 21:57:58.763953 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.763914 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" Jul 17 21:57:58.763953 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.763924 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" Jul 17 21:57:58.763953 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.763943 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" Jul 17 21:57:58.763953 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.763948 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" Jul 17 21:57:58.764097 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.763999 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" Jul 17 21:57:58.764097 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.764008 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c75f93-c142-469e-a289-34743fabf496" containerName="istio-proxy" Jul 17 21:57:58.768614 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.768592 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.773036 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.773014 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e3cf3d26-openshift-default-dockercfg-5ncg7\"" Jul 17 21:57:58.783064 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.783035 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf"] Jul 17 21:57:58.891544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891508 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-token\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891544 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891542 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6tn\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-kube-api-access-rt6tn\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891791 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891563 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-socket\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891791 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891677 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-credential-socket\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891791 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-podinfo\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891908 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-certs\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891908 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891844 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-envoy\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891908 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891876 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-data\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.891908 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.891901 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istiod-ca-cert\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993276 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993212 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-envoy\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993433 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993309 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-data\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993433 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istiod-ca-cert\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993433 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993393 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-token\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993433 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6tn\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-kube-api-access-rt6tn\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993433 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-socket\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993707 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993684 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-data\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993766 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-credential-socket\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993766 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993755 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-podinfo\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993876 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993770 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-socket\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.993876 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.993839 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-certs\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.994023 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.994002 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-credential-socket\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.994125 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.994103 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-certs\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.994217 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.994130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istiod-ca-cert\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.995742 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.995718 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-envoy\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:58.996151 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:58.996134 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-podinfo\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:59.007841 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.007777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-token\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:59.007934 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.007917 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6tn\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-kube-api-access-rt6tn\") pod \"e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:59.079253 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.079190 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:57:59.225214 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.225186 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf"] Jul 17 21:57:59.227591 ip-10-0-139-178 kubenswrapper[2566]: W0717 21:57:59.227565 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3f6f51_efd2_4f9d_b0fc_edc86d6105b6.slice/crio-266daa3df729c20088966f94e93adeab9d5365280e16a7297a42d3233fe91896 WatchSource:0}: Error finding container 266daa3df729c20088966f94e93adeab9d5365280e16a7297a42d3233fe91896: Status 404 returned error can't find the container with id 266daa3df729c20088966f94e93adeab9d5365280e16a7297a42d3233fe91896 Jul 17 21:57:59.229662 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.229631 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:57:59.229724 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.229702 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:57:59.229762 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.229730 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:57:59.798949 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.798911 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" event={"ID":"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6","Type":"ContainerStarted","Data":"c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03"} Jul 17 21:57:59.799364 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.798960 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" event={"ID":"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6","Type":"ContainerStarted","Data":"266daa3df729c20088966f94e93adeab9d5365280e16a7297a42d3233fe91896"} Jul 17 21:57:59.822702 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:57:59.822653 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" podStartSLOduration=1.8226385170000001 podStartE2EDuration="1.822638517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:59.820071716 +0000 UTC m=+1817.090069447" watchObservedRunningTime="2026-07-17 21:57:59.822638517 +0000 UTC m=+1817.092636220" Jul 17 21:58:00.080346 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:00.080256 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:58:00.085367 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:00.085342 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:58:00.803762 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:00.803728 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:58:00.804845 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:00.804812 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:58:02.810559 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:02.810520 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 21:58:11.860331 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:11.860293 2566 generic.go:356] "Generic (PLEG): container finished" podID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerID="c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03" exitCode=0 Jul 17 21:58:11.860804 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:11.860365 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" event={"ID":"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6","Type":"ContainerDied","Data":"c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03"} Jul 17 21:58:11.860804 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:11.860785 2566 scope.go:117] "RemoveContainer" containerID="c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03" Jul 17 21:58:11.861819 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:11.861789 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:58:11.861915 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:11.861871 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:58:11.861980 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:11.861919 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:58:12.872408 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:12.872368 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" event={"ID":"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6","Type":"ContainerStarted","Data":"89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4"} Jul 17 21:58:13.080115 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:13.080080 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:58:13.081405 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:13.081380 2566 patch_prober.go:28] interesting pod/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 21:58:13.081507 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:13.081440 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 21:58:14.080272 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:14.080214 2566 patch_prober.go:28] interesting pod/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 21:58:14.080653 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:14.080292 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 21:58:15.079811 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:15.079767 2566 patch_prober.go:28] interesting pod/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 21:58:15.079993 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:15.079834 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 21:58:15.803664 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:15.803605 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:58:16.080620 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:16.080523 2566 patch_prober.go:28] interesting pod/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 17 21:58:16.080620 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:16.080594 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 17 21:58:17.083520 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:17.083464 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 21:58:17.084533 ip-10-0-139-178 kubenswrapper[2566]: I0717 21:58:17.084510 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 22:04:05.236101 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:05.236064 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf"] Jul 17 22:04:05.236688 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:05.236305 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" containerID="cri-o://89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4" gracePeriod=30 Jul 17 22:04:08.103326 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:08.103297 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:04:10.495516 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.495489 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 22:04:10.508592 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508525 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-podinfo\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508592 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508563 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-data\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508592 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508592 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-certs\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508896 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508624 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt6tn\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-kube-api-access-rt6tn\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508896 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508658 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-envoy\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508896 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508710 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-socket\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508896 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508745 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-token\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508896 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508785 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-credential-socket\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508896 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508818 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istiod-ca-cert\") pod \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\" (UID: \"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6\") " Jul 17 22:04:10.508896 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.508882 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:04:10.509294 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.509120 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.509294 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.509145 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:04:10.509294 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.509212 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:04:10.509470 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.509413 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:04:10.509549 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.509530 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:04:10.511036 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.511004 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-kube-api-access-rt6tn" (OuterVolumeSpecName: "kube-api-access-rt6tn") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "kube-api-access-rt6tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:04:10.511441 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.511416 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:04:10.511540 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.511414 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:04:10.511540 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.511520 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" (UID: "8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:04:10.579737 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.579703 2566 generic.go:356] "Generic (PLEG): container finished" podID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerID="89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4" exitCode=0 Jul 17 22:04:10.580022 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.579749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" event={"ID":"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6","Type":"ContainerDied","Data":"89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4"} Jul 17 22:04:10.580022 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.579772 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" Jul 17 22:04:10.580022 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.579829 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf" event={"ID":"8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6","Type":"ContainerDied","Data":"266daa3df729c20088966f94e93adeab9d5365280e16a7297a42d3233fe91896"} Jul 17 22:04:10.580022 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.579852 2566 scope.go:117] "RemoveContainer" containerID="89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4" Jul 17 22:04:10.589898 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.589881 2566 scope.go:117] "RemoveContainer" containerID="c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03" Jul 17 22:04:10.598545 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.598528 2566 scope.go:117] "RemoveContainer" containerID="89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4" Jul 17 22:04:10.598811 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:04:10.598789 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4\": container with ID starting with 89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4 not found: ID does not exist" containerID="89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4" Jul 17 22:04:10.598894 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.598822 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4"} err="failed to get container status \"89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4\": rpc error: code = NotFound desc = could not find container \"89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4\": container with ID starting with 89f707ec4d96e3ec8d30344dee8cd8c34635f9cdd48f52a37aac9b0591e0bbc4 not found: ID does not exist" Jul 17 22:04:10.598894 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.598845 2566 scope.go:117] "RemoveContainer" containerID="c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03" Jul 17 22:04:10.599135 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:04:10.599116 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03\": container with ID starting with c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03 not found: ID does not exist" containerID="c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03" Jul 17 22:04:10.599184 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.599142 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03"} err="failed to get container status \"c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03\": rpc error: code = NotFound desc = could not find container \"c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03\": container with ID starting with c3336235bb76970e614d094760b74fdb92ae28c108bc99ef66942e1850ebaa03 not found: ID does not exist" Jul 17 22:04:10.606779 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.606743 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf"] Jul 17 22:04:10.609710 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609683 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt6tn\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-kube-api-access-rt6tn\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.609710 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609707 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.609861 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609724 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.609861 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609735 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.609861 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609748 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.609861 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609759 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.609861 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609771 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.609861 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.609783 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:10.610088 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:10.610068 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e3cf3d26-openshift-default-795d4b8d5f-phrmf"] Jul 17 22:04:11.299468 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:11.299434 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" path="/var/lib/kubelet/pods/8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6/volumes" Jul 17 22:04:15.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.299511 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5"] Jul 17 22:04:15.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.300317 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" Jul 17 22:04:15.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.300337 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" Jul 17 22:04:15.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.300357 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" Jul 17 22:04:15.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.300365 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" Jul 17 22:04:15.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.300527 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" Jul 17 22:04:15.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.300547 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3f6f51-efd2-4f9d-b0fc-edc86d6105b6" containerName="istio-proxy" Jul 17 22:04:15.306102 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.306073 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" Jul 17 22:04:15.308298 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.308269 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5"] Jul 17 22:04:15.309013 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.308990 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:04:15.309139 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.309013 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:04:15.351624 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.351582 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65b19b-64d7-498d-bae4-1959195b1ddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5\" (UID: \"fe65b19b-64d7-498d-bae4-1959195b1ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" Jul 17 22:04:15.452281 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.452243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65b19b-64d7-498d-bae4-1959195b1ddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5\" (UID: \"fe65b19b-64d7-498d-bae4-1959195b1ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" Jul 17 22:04:15.452874 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.452852 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65b19b-64d7-498d-bae4-1959195b1ddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5\" (UID: \"fe65b19b-64d7-498d-bae4-1959195b1ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" Jul 17 22:04:15.616512 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.616428 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" Jul 17 22:04:15.751536 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:15.751509 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5"] Jul 17 22:04:15.754335 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:04:15.754302 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe65b19b_64d7_498d_bae4_1959195b1ddb.slice/crio-fa4affa1a567fb9ea7df6338d1e23a26f9b32b1843258893cae8fccb4d3c5ba2 WatchSource:0}: Error finding container fa4affa1a567fb9ea7df6338d1e23a26f9b32b1843258893cae8fccb4d3c5ba2: Status 404 returned error can't find the container with id fa4affa1a567fb9ea7df6338d1e23a26f9b32b1843258893cae8fccb4d3c5ba2 Jul 17 22:04:16.611200 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:16.611164 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerID="d59278a763050c8846124959038bbe14fa54ca8f3b45e4c40002bbb0c4dfaf47" exitCode=6 Jul 17 22:04:16.611200 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:16.611201 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" event={"ID":"fe65b19b-64d7-498d-bae4-1959195b1ddb","Type":"ContainerDied","Data":"d59278a763050c8846124959038bbe14fa54ca8f3b45e4c40002bbb0c4dfaf47"} Jul 17 22:04:16.611729 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:16.611257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" event={"ID":"fe65b19b-64d7-498d-bae4-1959195b1ddb","Type":"ContainerStarted","Data":"fa4affa1a567fb9ea7df6338d1e23a26f9b32b1843258893cae8fccb4d3c5ba2"} Jul 17 22:04:16.611729 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:16.611506 2566 scope.go:117] "RemoveContainer" containerID="d59278a763050c8846124959038bbe14fa54ca8f3b45e4c40002bbb0c4dfaf47" Jul 17 22:04:17.236301 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.236273 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk"] Jul 17 22:04:17.239578 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.239563 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.243758 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.243735 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6f4935d4-openshift-default-dockercfg-hsws4\"" Jul 17 22:04:17.255013 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.254988 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk"] Jul 17 22:04:17.269708 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269678 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-certs\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269831 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-envoy\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269831 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-data\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269831 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e733d0-5432-401a-a85f-01fb90782bc8-istiod-ca-cert\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269831 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269794 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-socket\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269831 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-istio-token\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269998 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269873 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e733d0-5432-401a-a85f-01fb90782bc8-istio-podinfo\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269998 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269918 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-credential-socket\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.269998 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.269938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tc6\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-kube-api-access-66tc6\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.370991 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.370959 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-certs\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371149 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371005 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-envoy\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371149 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371045 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-data\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371149 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e733d0-5432-401a-a85f-01fb90782bc8-istiod-ca-cert\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371149 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-socket\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371149 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-istio-token\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371438 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371166 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e733d0-5432-401a-a85f-01fb90782bc8-istio-podinfo\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371438 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371195 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-credential-socket\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371438 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371219 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66tc6\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-kube-api-access-66tc6\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371438 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371388 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-certs\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371630 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371442 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-data\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371742 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371722 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-credential-socket\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371790 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-socket\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.371836 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.371820 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e733d0-5432-401a-a85f-01fb90782bc8-istiod-ca-cert\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.373425 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.373405 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-envoy\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.373685 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.373664 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e733d0-5432-401a-a85f-01fb90782bc8-istio-podinfo\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.379837 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.379814 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-istio-token\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.380120 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.380104 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tc6\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-kube-api-access-66tc6\") pod \"e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.551309 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.551186 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:17.620933 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.620903 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerID="4078d405ca082b5564e262abdc4ba6e52d89f1139bb81fb09d50114dba7e14be" exitCode=6 Jul 17 22:04:17.621308 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.620962 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" event={"ID":"fe65b19b-64d7-498d-bae4-1959195b1ddb","Type":"ContainerDied","Data":"4078d405ca082b5564e262abdc4ba6e52d89f1139bb81fb09d50114dba7e14be"} Jul 17 22:04:17.621308 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.621017 2566 scope.go:117] "RemoveContainer" containerID="d59278a763050c8846124959038bbe14fa54ca8f3b45e4c40002bbb0c4dfaf47" Jul 17 22:04:17.621422 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.621322 2566 scope.go:117] "RemoveContainer" containerID="4078d405ca082b5564e262abdc4ba6e52d89f1139bb81fb09d50114dba7e14be" Jul 17 22:04:17.621609 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:04:17.621579 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5_odh-ai-gateway-infra(fe65b19b-64d7-498d-bae4-1959195b1ddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" Jul 17 22:04:17.703058 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.703032 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk"] Jul 17 22:04:17.705069 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:04:17.704994 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e733d0_5432_401a_a85f_01fb90782bc8.slice/crio-e96e0e52393300ef7399fe03c0414d2967c5f8227424bf8b53d2d561ac67872d WatchSource:0}: Error finding container e96e0e52393300ef7399fe03c0414d2967c5f8227424bf8b53d2d561ac67872d: Status 404 returned error can't find the container with id e96e0e52393300ef7399fe03c0414d2967c5f8227424bf8b53d2d561ac67872d Jul 17 22:04:17.709739 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.709701 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:04:17.709829 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.709788 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:04:17.709877 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:17.709827 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:04:18.627196 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:18.627163 2566 scope.go:117] "RemoveContainer" containerID="4078d405ca082b5564e262abdc4ba6e52d89f1139bb81fb09d50114dba7e14be" Jul 17 22:04:18.627619 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:04:18.627455 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5_odh-ai-gateway-infra(fe65b19b-64d7-498d-bae4-1959195b1ddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" Jul 17 22:04:18.628464 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:18.628438 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" event={"ID":"72e733d0-5432-401a-a85f-01fb90782bc8","Type":"ContainerStarted","Data":"92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e"} Jul 17 22:04:18.628580 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:18.628491 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" event={"ID":"72e733d0-5432-401a-a85f-01fb90782bc8","Type":"ContainerStarted","Data":"e96e0e52393300ef7399fe03c0414d2967c5f8227424bf8b53d2d561ac67872d"} Jul 17 22:04:18.663321 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:18.663271 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" podStartSLOduration=1.6632535659999999 podStartE2EDuration="1.663253566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:04:18.661493388 +0000 UTC m=+2195.931491118" watchObservedRunningTime="2026-07-17 22:04:18.663253566 +0000 UTC m=+2195.933251332" Jul 17 22:04:19.552317 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:19.552271 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:19.557404 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:19.557378 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:19.634250 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:19.634212 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:19.635330 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:19.635308 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:21.207656 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:21.207615 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:04:29.704327 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:29.704289 2566 generic.go:356] "Generic (PLEG): container finished" podID="72e733d0-5432-401a-a85f-01fb90782bc8" containerID="92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e" exitCode=0 Jul 17 22:04:29.704768 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:29.704365 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" event={"ID":"72e733d0-5432-401a-a85f-01fb90782bc8","Type":"ContainerDied","Data":"92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e"} Jul 17 22:04:29.704817 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:29.704779 2566 scope.go:117] "RemoveContainer" containerID="92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e" Jul 17 22:04:29.705746 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:29.705714 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:04:29.705839 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:29.705800 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:04:29.705907 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:29.705865 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:04:30.710907 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:30.710874 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" event={"ID":"72e733d0-5432-401a-a85f-01fb90782bc8","Type":"ContainerStarted","Data":"7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9"} Jul 17 22:04:31.552387 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:31.552350 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:31.553330 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:31.553283 2566 patch_prober.go:28] interesting pod/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 17 22:04:31.553481 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:31.553362 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 17 22:04:32.552713 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:32.552662 2566 patch_prober.go:28] interesting pod/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 17 22:04:32.553217 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:32.552737 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 17 22:04:33.295132 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:33.294975 2566 scope.go:117] "RemoveContainer" containerID="4078d405ca082b5564e262abdc4ba6e52d89f1139bb81fb09d50114dba7e14be" Jul 17 22:04:33.552435 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:33.552398 2566 patch_prober.go:28] interesting pod/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 17 22:04:33.552626 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:33.552485 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 17 22:04:33.726804 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:33.726763 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" event={"ID":"fe65b19b-64d7-498d-bae4-1959195b1ddb","Type":"ContainerStarted","Data":"7f660e307b7b8182e2aef8d9524ed40f22c5fc89c4d78e063c5c880861f2d1ae"} Jul 17 22:04:33.744593 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:33.744544 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" podStartSLOduration=18.744528435 podStartE2EDuration="18.744528435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:04:33.743150087 +0000 UTC m=+2211.013147792" watchObservedRunningTime="2026-07-17 22:04:33.744528435 +0000 UTC m=+2211.014526140" Jul 17 22:04:34.556722 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.556635 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:34.557160 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.557144 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:34.558001 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.557977 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:04:34.733191 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.733158 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerID="7f660e307b7b8182e2aef8d9524ed40f22c5fc89c4d78e063c5c880861f2d1ae" exitCode=6 Jul 17 22:04:34.733626 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.733258 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" event={"ID":"fe65b19b-64d7-498d-bae4-1959195b1ddb","Type":"ContainerDied","Data":"7f660e307b7b8182e2aef8d9524ed40f22c5fc89c4d78e063c5c880861f2d1ae"} Jul 17 22:04:34.733626 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.733316 2566 scope.go:117] "RemoveContainer" containerID="4078d405ca082b5564e262abdc4ba6e52d89f1139bb81fb09d50114dba7e14be" Jul 17 22:04:34.733784 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.733763 2566 scope.go:117] "RemoveContainer" containerID="7f660e307b7b8182e2aef8d9524ed40f22c5fc89c4d78e063c5c880861f2d1ae" Jul 17 22:04:34.733982 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:04:34.733962 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5_odh-ai-gateway-infra(fe65b19b-64d7-498d-bae4-1959195b1ddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" Jul 17 22:04:34.754908 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:34.754882 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5"] Jul 17 22:04:35.884852 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:35.884827 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" Jul 17 22:04:35.945150 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:35.945114 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65b19b-64d7-498d-bae4-1959195b1ddb-maas-api-service-ca\") pod \"fe65b19b-64d7-498d-bae4-1959195b1ddb\" (UID: \"fe65b19b-64d7-498d-bae4-1959195b1ddb\") " Jul 17 22:04:35.945462 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:35.945439 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe65b19b-64d7-498d-bae4-1959195b1ddb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe65b19b-64d7-498d-bae4-1959195b1ddb" (UID: "fe65b19b-64d7-498d-bae4-1959195b1ddb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:04:36.046834 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:36.046741 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe65b19b-64d7-498d-bae4-1959195b1ddb-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:04:36.750433 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:36.750405 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" Jul 17 22:04:36.750697 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:36.750440 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5" event={"ID":"fe65b19b-64d7-498d-bae4-1959195b1ddb","Type":"ContainerDied","Data":"fa4affa1a567fb9ea7df6338d1e23a26f9b32b1843258893cae8fccb4d3c5ba2"} Jul 17 22:04:36.750697 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:36.750494 2566 scope.go:117] "RemoveContainer" containerID="7f660e307b7b8182e2aef8d9524ed40f22c5fc89c4d78e063c5c880861f2d1ae" Jul 17 22:04:36.773557 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:36.773525 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5"] Jul 17 22:04:36.779810 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:36.779787 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-c2fq5"] Jul 17 22:04:37.297892 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:04:37.297862 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" path="/var/lib/kubelet/pods/fe65b19b-64d7-498d-bae4-1959195b1ddb/volumes" Jul 17 22:07:22.927565 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:22.927530 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk"] Jul 17 22:07:22.928140 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:22.927805 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" containerID="cri-o://7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9" gracePeriod=30 Jul 17 22:07:25.906388 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:25.906355 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:07:28.191710 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.191648 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:07:28.201369 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201346 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-envoy\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201483 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201376 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-istio-token\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201483 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201406 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e733d0-5432-401a-a85f-01fb90782bc8-istio-podinfo\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201483 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201449 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-socket\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201483 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201479 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-certs\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201689 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201535 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66tc6\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-kube-api-access-66tc6\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201689 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201605 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-data\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201689 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201629 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e733d0-5432-401a-a85f-01fb90782bc8-istiod-ca-cert\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201689 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201671 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-credential-socket\") pod \"72e733d0-5432-401a-a85f-01fb90782bc8\" (UID: \"72e733d0-5432-401a-a85f-01fb90782bc8\") " Jul 17 22:07:28.201892 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201784 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:28.201892 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.201812 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:28.202042 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.202019 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:28.202113 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.202042 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.202169 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.202119 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e733d0-5432-401a-a85f-01fb90782bc8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:07:28.202169 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.202108 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.202689 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.202657 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-data" (OuterVolumeSpecName: "istio-data") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:28.203858 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.203831 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-istio-token" (OuterVolumeSpecName: "istio-token") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:07:28.204192 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.204165 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-kube-api-access-66tc6" (OuterVolumeSpecName: "kube-api-access-66tc6") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "kube-api-access-66tc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:07:28.204307 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.204269 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72e733d0-5432-401a-a85f-01fb90782bc8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:07:28.204363 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.204317 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72e733d0-5432-401a-a85f-01fb90782bc8" (UID: "72e733d0-5432-401a-a85f-01fb90782bc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:07:28.303053 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.303023 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66tc6\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-kube-api-access-66tc6\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.303053 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.303053 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.303053 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.303062 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e733d0-5432-401a-a85f-01fb90782bc8-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.303284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.303069 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.303284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.303078 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e733d0-5432-401a-a85f-01fb90782bc8-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.303284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.303086 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e733d0-5432-401a-a85f-01fb90782bc8-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.303284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.303093 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e733d0-5432-401a-a85f-01fb90782bc8-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:28.578415 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.578323 2566 generic.go:356] "Generic (PLEG): container finished" podID="72e733d0-5432-401a-a85f-01fb90782bc8" containerID="7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9" exitCode=0 Jul 17 22:07:28.578415 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.578370 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" Jul 17 22:07:28.578654 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.578405 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" event={"ID":"72e733d0-5432-401a-a85f-01fb90782bc8","Type":"ContainerDied","Data":"7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9"} Jul 17 22:07:28.578654 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.578476 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk" event={"ID":"72e733d0-5432-401a-a85f-01fb90782bc8","Type":"ContainerDied","Data":"e96e0e52393300ef7399fe03c0414d2967c5f8227424bf8b53d2d561ac67872d"} Jul 17 22:07:28.578654 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.578507 2566 scope.go:117] "RemoveContainer" containerID="7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9" Jul 17 22:07:28.588108 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.588084 2566 scope.go:117] "RemoveContainer" containerID="92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e" Jul 17 22:07:28.597209 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.597186 2566 scope.go:117] "RemoveContainer" containerID="7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9" Jul 17 22:07:28.597546 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:07:28.597517 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9\": container with ID starting with 7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9 not found: ID does not exist" containerID="7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9" Jul 17 22:07:28.597620 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.597558 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9"} err="failed to get container status \"7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9\": rpc error: code = NotFound desc = could not find container \"7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9\": container with ID starting with 7b3b57919f3f575cded143dd4a891e9a7f61a1886fc78a3d12e7a668308a7bc9 not found: ID does not exist" Jul 17 22:07:28.597620 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.597585 2566 scope.go:117] "RemoveContainer" containerID="92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e" Jul 17 22:07:28.597876 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:07:28.597856 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e\": container with ID starting with 92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e not found: ID does not exist" containerID="92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e" Jul 17 22:07:28.597921 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.597884 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e"} err="failed to get container status \"92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e\": rpc error: code = NotFound desc = could not find container \"92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e\": container with ID starting with 92587592c3ef9fd9b5c37d2721affe47edb36014cc678d21ad7c8d64bb1dd96e not found: ID does not exist" Jul 17 22:07:28.611201 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.611175 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk"] Jul 17 22:07:28.613845 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:28.613819 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6f4935d4-openshift-default-55564ccc79-8x8jk"] Jul 17 22:07:29.298649 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.298614 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" path="/var/lib/kubelet/pods/72e733d0-5432-401a-a85f-01fb90782bc8/volumes" Jul 17 22:07:29.779307 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779268 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5"] Jul 17 22:07:29.779638 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779625 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779683 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779639 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779683 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779650 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" Jul 17 22:07:29.779683 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779656 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" Jul 17 22:07:29.779683 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779663 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779683 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779668 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779683 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779682 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" Jul 17 22:07:29.779872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779688 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" Jul 17 22:07:29.779872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779698 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779703 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779755 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779763 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779771 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe65b19b-64d7-498d-bae4-1959195b1ddb" containerName="revoke-keys" Jul 17 22:07:29.779872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.779778 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" Jul 17 22:07:29.784242 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.784211 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" Jul 17 22:07:29.786832 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.786796 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:07:29.786832 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.786818 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:07:29.790682 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.790658 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5"] Jul 17 22:07:29.815893 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.815855 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5\" (UID: \"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" Jul 17 22:07:29.916756 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.916707 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5\" (UID: \"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" Jul 17 22:07:29.917398 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:29.917380 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5\" (UID: \"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" Jul 17 22:07:30.096175 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:30.096087 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" Jul 17 22:07:30.231223 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:30.231200 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5"] Jul 17 22:07:30.233894 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:07:30.233868 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce606ae_aa62_40d4_a6ab_dc699fbf3d5c.slice/crio-6518766ee186ca475e424ec352d264f938c91b21ee01a8c159a92357c3c33c64 WatchSource:0}: Error finding container 6518766ee186ca475e424ec352d264f938c91b21ee01a8c159a92357c3c33c64: Status 404 returned error can't find the container with id 6518766ee186ca475e424ec352d264f938c91b21ee01a8c159a92357c3c33c64 Jul 17 22:07:30.591170 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:30.591136 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" event={"ID":"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c","Type":"ContainerStarted","Data":"8955d11261aff1d7c5906f458b5adf5a6221e7cecb3b5485f9c4600da1430074"} Jul 17 22:07:30.591544 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:30.591186 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" event={"ID":"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c","Type":"ContainerStarted","Data":"6518766ee186ca475e424ec352d264f938c91b21ee01a8c159a92357c3c33c64"} Jul 17 22:07:30.613045 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:30.612995 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" podStartSLOduration=1.6129795900000001 podStartE2EDuration="1.61297959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:30.612147441 +0000 UTC m=+2387.882145170" watchObservedRunningTime="2026-07-17 22:07:30.61297959 +0000 UTC m=+2387.882977294" Jul 17 22:07:31.597128 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:31.597089 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerID="8955d11261aff1d7c5906f458b5adf5a6221e7cecb3b5485f9c4600da1430074" exitCode=6 Jul 17 22:07:31.597554 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:31.597154 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" event={"ID":"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c","Type":"ContainerDied","Data":"8955d11261aff1d7c5906f458b5adf5a6221e7cecb3b5485f9c4600da1430074"} Jul 17 22:07:31.597554 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:31.597378 2566 scope.go:117] "RemoveContainer" containerID="8955d11261aff1d7c5906f458b5adf5a6221e7cecb3b5485f9c4600da1430074" Jul 17 22:07:32.604548 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:32.604458 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerID="ec90eaa574b0a8d399e2c65217500ba264de06ed1e9b172d58350f4c8c7e8230" exitCode=6 Jul 17 22:07:32.604548 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:32.604507 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" event={"ID":"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c","Type":"ContainerDied","Data":"ec90eaa574b0a8d399e2c65217500ba264de06ed1e9b172d58350f4c8c7e8230"} Jul 17 22:07:32.605004 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:32.604569 2566 scope.go:117] "RemoveContainer" containerID="8955d11261aff1d7c5906f458b5adf5a6221e7cecb3b5485f9c4600da1430074" Jul 17 22:07:32.605004 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:32.604813 2566 scope.go:117] "RemoveContainer" containerID="ec90eaa574b0a8d399e2c65217500ba264de06ed1e9b172d58350f4c8c7e8230" Jul 17 22:07:32.605085 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:07:32.605025 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5_odh-ai-gateway-infra(0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" Jul 17 22:07:33.613220 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:33.613193 2566 scope.go:117] "RemoveContainer" containerID="ec90eaa574b0a8d399e2c65217500ba264de06ed1e9b172d58350f4c8c7e8230" Jul 17 22:07:33.613617 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:07:33.613424 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5_odh-ai-gateway-infra(0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" Jul 17 22:07:46.292787 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:46.292748 2566 scope.go:117] "RemoveContainer" containerID="ec90eaa574b0a8d399e2c65217500ba264de06ed1e9b172d58350f4c8c7e8230" Jul 17 22:07:46.677331 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:46.677296 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" event={"ID":"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c","Type":"ContainerStarted","Data":"54a59b657597c72f7ae807d8feca90430a04583892c0cb32f2a111004d6f9543"} Jul 17 22:07:47.631440 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.631405 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz"] Jul 17 22:07:47.631906 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.631819 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e733d0-5432-401a-a85f-01fb90782bc8" containerName="istio-proxy" Jul 17 22:07:47.634557 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.634533 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" Jul 17 22:07:47.643714 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.643689 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz"] Jul 17 22:07:47.684253 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.684190 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerID="54a59b657597c72f7ae807d8feca90430a04583892c0cb32f2a111004d6f9543" exitCode=6 Jul 17 22:07:47.684435 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.684262 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" event={"ID":"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c","Type":"ContainerDied","Data":"54a59b657597c72f7ae807d8feca90430a04583892c0cb32f2a111004d6f9543"} Jul 17 22:07:47.684435 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.684329 2566 scope.go:117] "RemoveContainer" containerID="ec90eaa574b0a8d399e2c65217500ba264de06ed1e9b172d58350f4c8c7e8230" Jul 17 22:07:47.684593 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.684579 2566 scope.go:117] "RemoveContainer" containerID="54a59b657597c72f7ae807d8feca90430a04583892c0cb32f2a111004d6f9543" Jul 17 22:07:47.684853 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:07:47.684831 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5_odh-ai-gateway-infra(0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" Jul 17 22:07:47.706989 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.706963 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5"] Jul 17 22:07:47.782346 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.782310 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceda3d4-760c-458f-8b8c-666bfc458d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz\" (UID: \"fceda3d4-760c-458f-8b8c-666bfc458d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" Jul 17 22:07:47.883729 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.883643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceda3d4-760c-458f-8b8c-666bfc458d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz\" (UID: \"fceda3d4-760c-458f-8b8c-666bfc458d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" Jul 17 22:07:47.884321 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.884300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceda3d4-760c-458f-8b8c-666bfc458d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz\" (UID: \"fceda3d4-760c-458f-8b8c-666bfc458d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" Jul 17 22:07:47.946720 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:47.946689 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" Jul 17 22:07:48.093260 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.093203 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz"] Jul 17 22:07:48.097004 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:07:48.096970 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceda3d4_760c_458f_8b8c_666bfc458d33.slice/crio-a4d132e5d8f61dbc23159e02658d7d1f7209637587c4e8716554b4d3a8bbba9d WatchSource:0}: Error finding container a4d132e5d8f61dbc23159e02658d7d1f7209637587c4e8716554b4d3a8bbba9d: Status 404 returned error can't find the container with id a4d132e5d8f61dbc23159e02658d7d1f7209637587c4e8716554b4d3a8bbba9d Jul 17 22:07:48.692395 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.692362 2566 generic.go:356] "Generic (PLEG): container finished" podID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerID="9ee08830103fea3a42a4e97c7cf62a2a7a8e7ce91f4f9a980e2ca1a7fe9668fa" exitCode=6 Jul 17 22:07:48.692734 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.692447 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" event={"ID":"fceda3d4-760c-458f-8b8c-666bfc458d33","Type":"ContainerDied","Data":"9ee08830103fea3a42a4e97c7cf62a2a7a8e7ce91f4f9a980e2ca1a7fe9668fa"} Jul 17 22:07:48.692734 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.692516 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" event={"ID":"fceda3d4-760c-458f-8b8c-666bfc458d33","Type":"ContainerStarted","Data":"a4d132e5d8f61dbc23159e02658d7d1f7209637587c4e8716554b4d3a8bbba9d"} Jul 17 22:07:48.692734 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.692605 2566 scope.go:117] "RemoveContainer" containerID="9ee08830103fea3a42a4e97c7cf62a2a7a8e7ce91f4f9a980e2ca1a7fe9668fa" Jul 17 22:07:48.808844 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.808819 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" Jul 17 22:07:48.894749 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.894712 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c-maas-api-service-ca\") pod \"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c\" (UID: \"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c\") " Jul 17 22:07:48.895117 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.895095 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" (UID: "0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:07:48.996486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:48.996450 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:07:49.698825 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.698794 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" Jul 17 22:07:49.699336 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.698837 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5" event={"ID":"0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c","Type":"ContainerDied","Data":"6518766ee186ca475e424ec352d264f938c91b21ee01a8c159a92357c3c33c64"} Jul 17 22:07:49.699336 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.698903 2566 scope.go:117] "RemoveContainer" containerID="54a59b657597c72f7ae807d8feca90430a04583892c0cb32f2a111004d6f9543" Jul 17 22:07:49.701088 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.701060 2566 generic.go:356] "Generic (PLEG): container finished" podID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerID="5b7d18f0c29d9e2c42c8ebddefdcebe250b3ef614f27b190f42e20c196584849" exitCode=6 Jul 17 22:07:49.701213 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.701101 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" event={"ID":"fceda3d4-760c-458f-8b8c-666bfc458d33","Type":"ContainerDied","Data":"5b7d18f0c29d9e2c42c8ebddefdcebe250b3ef614f27b190f42e20c196584849"} Jul 17 22:07:49.701424 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.701408 2566 scope.go:117] "RemoveContainer" containerID="5b7d18f0c29d9e2c42c8ebddefdcebe250b3ef614f27b190f42e20c196584849" Jul 17 22:07:49.701629 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:07:49.701608 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz_odh-ai-gateway-infra(fceda3d4-760c-458f-8b8c-666bfc458d33)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" Jul 17 22:07:49.707964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.707946 2566 scope.go:117] "RemoveContainer" containerID="9ee08830103fea3a42a4e97c7cf62a2a7a8e7ce91f4f9a980e2ca1a7fe9668fa" Jul 17 22:07:49.718869 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.718842 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5"] Jul 17 22:07:49.726818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:49.726795 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-9nwk5"] Jul 17 22:07:50.707537 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:50.707504 2566 scope.go:117] "RemoveContainer" containerID="5b7d18f0c29d9e2c42c8ebddefdcebe250b3ef614f27b190f42e20c196584849" Jul 17 22:07:50.707915 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:07:50.707727 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz_odh-ai-gateway-infra(fceda3d4-760c-458f-8b8c-666bfc458d33)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" Jul 17 22:07:51.299161 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:07:51.299126 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" path="/var/lib/kubelet/pods/0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c/volumes" Jul 17 22:08:03.298412 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:03.298384 2566 scope.go:117] "RemoveContainer" containerID="5b7d18f0c29d9e2c42c8ebddefdcebe250b3ef614f27b190f42e20c196584849" Jul 17 22:08:03.769690 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:03.769612 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" event={"ID":"fceda3d4-760c-458f-8b8c-666bfc458d33","Type":"ContainerStarted","Data":"135eb9fd9a3282df6ec636bedefb8eeb560f9aea61693220fccbb0e3edaf5317"} Jul 17 22:08:03.788662 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:03.788603 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" podStartSLOduration=16.788587545 podStartE2EDuration="16.788587545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:03.785736964 +0000 UTC m=+2421.055734669" watchObservedRunningTime="2026-07-17 22:08:03.788587545 +0000 UTC m=+2421.058585248" Jul 17 22:08:04.776477 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:04.776443 2566 generic.go:356] "Generic (PLEG): container finished" podID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerID="135eb9fd9a3282df6ec636bedefb8eeb560f9aea61693220fccbb0e3edaf5317" exitCode=6 Jul 17 22:08:04.776914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:04.776511 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" event={"ID":"fceda3d4-760c-458f-8b8c-666bfc458d33","Type":"ContainerDied","Data":"135eb9fd9a3282df6ec636bedefb8eeb560f9aea61693220fccbb0e3edaf5317"} Jul 17 22:08:04.776914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:04.776569 2566 scope.go:117] "RemoveContainer" containerID="5b7d18f0c29d9e2c42c8ebddefdcebe250b3ef614f27b190f42e20c196584849" Jul 17 22:08:04.776914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:04.776849 2566 scope.go:117] "RemoveContainer" containerID="135eb9fd9a3282df6ec636bedefb8eeb560f9aea61693220fccbb0e3edaf5317" Jul 17 22:08:04.777067 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:08:04.777047 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz_odh-ai-gateway-infra(fceda3d4-760c-458f-8b8c-666bfc458d33)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" Jul 17 22:08:04.804660 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:04.804626 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz"] Jul 17 22:08:05.919247 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:05.919202 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" Jul 17 22:08:05.953375 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:05.953348 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceda3d4-760c-458f-8b8c-666bfc458d33-maas-api-service-ca\") pod \"fceda3d4-760c-458f-8b8c-666bfc458d33\" (UID: \"fceda3d4-760c-458f-8b8c-666bfc458d33\") " Jul 17 22:08:05.953697 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:05.953673 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fceda3d4-760c-458f-8b8c-666bfc458d33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fceda3d4-760c-458f-8b8c-666bfc458d33" (UID: "fceda3d4-760c-458f-8b8c-666bfc458d33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:08:06.054151 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:06.054063 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceda3d4-760c-458f-8b8c-666bfc458d33-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:08:06.788274 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:06.788243 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" Jul 17 22:08:06.788274 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:06.788252 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz" event={"ID":"fceda3d4-760c-458f-8b8c-666bfc458d33","Type":"ContainerDied","Data":"a4d132e5d8f61dbc23159e02658d7d1f7209637587c4e8716554b4d3a8bbba9d"} Jul 17 22:08:06.788526 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:06.788319 2566 scope.go:117] "RemoveContainer" containerID="135eb9fd9a3282df6ec636bedefb8eeb560f9aea61693220fccbb0e3edaf5317" Jul 17 22:08:06.819960 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:06.819928 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz"] Jul 17 22:08:06.823678 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:06.823638 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-8g2kz"] Jul 17 22:08:07.298574 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:07.298541 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" path="/var/lib/kubelet/pods/fceda3d4-760c-458f-8b8c-666bfc458d33/volumes" Jul 17 22:08:37.669152 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669115 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6"] Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669490 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669502 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669517 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669523 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669533 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669539 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669546 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669551 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669557 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669563 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669571 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669577 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669627 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669637 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669647 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669655 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fceda3d4-760c-458f-8b8c-666bfc458d33" containerName="revoke-keys" Jul 17 22:08:37.669666 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.669662 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:37.672515 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.672501 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" Jul 17 22:08:37.675037 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.675014 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:08:37.675162 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.675018 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:08:37.680198 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.679825 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6"] Jul 17 22:08:37.711146 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.711112 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53e1b1-109c-4b3e-9dd2-2321e6babb5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6\" (UID: \"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" Jul 17 22:08:37.811823 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.811787 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53e1b1-109c-4b3e-9dd2-2321e6babb5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6\" (UID: \"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" Jul 17 22:08:37.812450 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.812432 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53e1b1-109c-4b3e-9dd2-2321e6babb5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6\" (UID: \"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" Jul 17 22:08:37.984117 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:37.984032 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" Jul 17 22:08:38.125887 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:38.125855 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6"] Jul 17 22:08:38.131665 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:08:38.131626 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b53e1b1_109c_4b3e_9dd2_2321e6babb5f.slice/crio-94a18f0f7c4e459d6ce277a57c026693949c43e8af8455ef030fbcad9e03edae WatchSource:0}: Error finding container 94a18f0f7c4e459d6ce277a57c026693949c43e8af8455ef030fbcad9e03edae: Status 404 returned error can't find the container with id 94a18f0f7c4e459d6ce277a57c026693949c43e8af8455ef030fbcad9e03edae Jul 17 22:08:38.941519 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:38.941484 2566 generic.go:356] "Generic (PLEG): container finished" podID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerID="759774d2fe68c259806d19e76367e118b2abc2fe4aba12f70d43da41f6c2df90" exitCode=6 Jul 17 22:08:38.941916 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:38.941568 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" event={"ID":"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f","Type":"ContainerDied","Data":"759774d2fe68c259806d19e76367e118b2abc2fe4aba12f70d43da41f6c2df90"} Jul 17 22:08:38.941916 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:38.941671 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" event={"ID":"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f","Type":"ContainerStarted","Data":"94a18f0f7c4e459d6ce277a57c026693949c43e8af8455ef030fbcad9e03edae"} Jul 17 22:08:38.941916 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:38.941765 2566 scope.go:117] "RemoveContainer" containerID="759774d2fe68c259806d19e76367e118b2abc2fe4aba12f70d43da41f6c2df90" Jul 17 22:08:39.948174 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:39.948142 2566 generic.go:356] "Generic (PLEG): container finished" podID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerID="4ba0009f71411101c7f29d193acae421d12a5b8bc34faf6562a9126fbdba35f6" exitCode=6 Jul 17 22:08:39.948685 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:39.948224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" event={"ID":"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f","Type":"ContainerDied","Data":"4ba0009f71411101c7f29d193acae421d12a5b8bc34faf6562a9126fbdba35f6"} Jul 17 22:08:39.948685 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:39.948322 2566 scope.go:117] "RemoveContainer" containerID="759774d2fe68c259806d19e76367e118b2abc2fe4aba12f70d43da41f6c2df90" Jul 17 22:08:39.948685 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:39.948542 2566 scope.go:117] "RemoveContainer" containerID="4ba0009f71411101c7f29d193acae421d12a5b8bc34faf6562a9126fbdba35f6" Jul 17 22:08:39.948843 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:08:39.948780 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6_odh-ai-gateway-infra(1b53e1b1-109c-4b3e-9dd2-2321e6babb5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" Jul 17 22:08:40.954991 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:40.954959 2566 scope.go:117] "RemoveContainer" containerID="4ba0009f71411101c7f29d193acae421d12a5b8bc34faf6562a9126fbdba35f6" Jul 17 22:08:40.955440 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:08:40.955262 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6_odh-ai-gateway-infra(1b53e1b1-109c-4b3e-9dd2-2321e6babb5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" Jul 17 22:08:45.581181 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.581141 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl"] Jul 17 22:08:45.581594 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.581574 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ce606ae-aa62-40d4-a6ab-dc699fbf3d5c" containerName="revoke-keys" Jul 17 22:08:45.585791 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.585772 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" Jul 17 22:08:45.592657 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.592621 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl"] Jul 17 22:08:45.680186 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.680153 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a45201-bf95-49f4-abbb-c951d614eeb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl\" (UID: \"f4a45201-bf95-49f4-abbb-c951d614eeb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" Jul 17 22:08:45.781393 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.781360 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a45201-bf95-49f4-abbb-c951d614eeb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl\" (UID: \"f4a45201-bf95-49f4-abbb-c951d614eeb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" Jul 17 22:08:45.782037 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.782009 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a45201-bf95-49f4-abbb-c951d614eeb1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl\" (UID: \"f4a45201-bf95-49f4-abbb-c951d614eeb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" Jul 17 22:08:45.897992 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:45.897911 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" Jul 17 22:08:46.035200 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:46.035176 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl"] Jul 17 22:08:46.037250 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:08:46.037195 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a45201_bf95_49f4_abbb_c951d614eeb1.slice/crio-01ddacb830af64cb63aff2605e3257b172845198c43a97dc3cf3f642d0912a23 WatchSource:0}: Error finding container 01ddacb830af64cb63aff2605e3257b172845198c43a97dc3cf3f642d0912a23: Status 404 returned error can't find the container with id 01ddacb830af64cb63aff2605e3257b172845198c43a97dc3cf3f642d0912a23 Jul 17 22:08:46.984416 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:46.984384 2566 generic.go:356] "Generic (PLEG): container finished" podID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerID="ed74bf248e12568914ddc36adbfb788028d930a5d3b7c1f1c87b561d170fdd9b" exitCode=6 Jul 17 22:08:46.984806 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:46.984431 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" event={"ID":"f4a45201-bf95-49f4-abbb-c951d614eeb1","Type":"ContainerDied","Data":"ed74bf248e12568914ddc36adbfb788028d930a5d3b7c1f1c87b561d170fdd9b"} Jul 17 22:08:46.984806 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:46.984479 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" event={"ID":"f4a45201-bf95-49f4-abbb-c951d614eeb1","Type":"ContainerStarted","Data":"01ddacb830af64cb63aff2605e3257b172845198c43a97dc3cf3f642d0912a23"} Jul 17 22:08:46.984806 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:46.984707 2566 scope.go:117] "RemoveContainer" containerID="ed74bf248e12568914ddc36adbfb788028d930a5d3b7c1f1c87b561d170fdd9b" Jul 17 22:08:47.991207 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:47.991168 2566 generic.go:356] "Generic (PLEG): container finished" podID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerID="a25b3d7edb8880dc519e237178f80a48e8dbd859ca9f84786999b515ab57e425" exitCode=6 Jul 17 22:08:47.991675 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:47.991260 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" event={"ID":"f4a45201-bf95-49f4-abbb-c951d614eeb1","Type":"ContainerDied","Data":"a25b3d7edb8880dc519e237178f80a48e8dbd859ca9f84786999b515ab57e425"} Jul 17 22:08:47.991675 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:47.991336 2566 scope.go:117] "RemoveContainer" containerID="ed74bf248e12568914ddc36adbfb788028d930a5d3b7c1f1c87b561d170fdd9b" Jul 17 22:08:47.991675 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:47.991629 2566 scope.go:117] "RemoveContainer" containerID="a25b3d7edb8880dc519e237178f80a48e8dbd859ca9f84786999b515ab57e425" Jul 17 22:08:47.991882 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:08:47.991861 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl_odh-ai-gateway-infra(f4a45201-bf95-49f4-abbb-c951d614eeb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" Jul 17 22:08:48.998537 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:48.998512 2566 scope.go:117] "RemoveContainer" containerID="a25b3d7edb8880dc519e237178f80a48e8dbd859ca9f84786999b515ab57e425" Jul 17 22:08:48.998958 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:08:48.998714 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl_odh-ai-gateway-infra(f4a45201-bf95-49f4-abbb-c951d614eeb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" Jul 17 22:08:54.292547 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:54.292517 2566 scope.go:117] "RemoveContainer" containerID="4ba0009f71411101c7f29d193acae421d12a5b8bc34faf6562a9126fbdba35f6" Jul 17 22:08:55.030119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:55.030084 2566 generic.go:356] "Generic (PLEG): container finished" podID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerID="5e3cab131fa4244da134f5cfa11e8d31067726ad3a633f6bce9905335db3e152" exitCode=6 Jul 17 22:08:55.030287 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:55.030130 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" event={"ID":"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f","Type":"ContainerDied","Data":"5e3cab131fa4244da134f5cfa11e8d31067726ad3a633f6bce9905335db3e152"} Jul 17 22:08:55.030287 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:55.030177 2566 scope.go:117] "RemoveContainer" containerID="4ba0009f71411101c7f29d193acae421d12a5b8bc34faf6562a9126fbdba35f6" Jul 17 22:08:55.030493 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:55.030469 2566 scope.go:117] "RemoveContainer" containerID="5e3cab131fa4244da134f5cfa11e8d31067726ad3a633f6bce9905335db3e152" Jul 17 22:08:55.030747 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:08:55.030728 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6_odh-ai-gateway-infra(1b53e1b1-109c-4b3e-9dd2-2321e6babb5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" Jul 17 22:08:56.056040 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:56.055977 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6"] Jul 17 22:08:56.185733 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:56.185707 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" Jul 17 22:08:56.285070 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:56.285016 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53e1b1-109c-4b3e-9dd2-2321e6babb5f-maas-api-service-ca\") pod \"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f\" (UID: \"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f\") " Jul 17 22:08:56.285407 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:56.285384 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b53e1b1-109c-4b3e-9dd2-2321e6babb5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" (UID: "1b53e1b1-109c-4b3e-9dd2-2321e6babb5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:08:56.386147 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:56.386050 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b53e1b1-109c-4b3e-9dd2-2321e6babb5f-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:08:57.042959 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:57.042915 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" event={"ID":"1b53e1b1-109c-4b3e-9dd2-2321e6babb5f","Type":"ContainerDied","Data":"94a18f0f7c4e459d6ce277a57c026693949c43e8af8455ef030fbcad9e03edae"} Jul 17 22:08:57.042959 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:57.042935 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6" Jul 17 22:08:57.043192 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:57.043000 2566 scope.go:117] "RemoveContainer" containerID="5e3cab131fa4244da134f5cfa11e8d31067726ad3a633f6bce9905335db3e152" Jul 17 22:08:57.066171 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:57.066130 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6"] Jul 17 22:08:57.072050 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:57.072025 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-fphz6"] Jul 17 22:08:57.301070 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:57.300991 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" path="/var/lib/kubelet/pods/1b53e1b1-109c-4b3e-9dd2-2321e6babb5f/volumes" Jul 17 22:08:57.503222 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:08:57.503188 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:09:03.294851 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:03.294823 2566 scope.go:117] "RemoveContainer" containerID="a25b3d7edb8880dc519e237178f80a48e8dbd859ca9f84786999b515ab57e425" Jul 17 22:09:04.079672 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:04.079639 2566 generic.go:356] "Generic (PLEG): container finished" podID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerID="09c33f4a31f5bda34a21e934d0d87540be0a2d9da3478410832b7dca869242a3" exitCode=6 Jul 17 22:09:04.079967 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:04.079682 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" event={"ID":"f4a45201-bf95-49f4-abbb-c951d614eeb1","Type":"ContainerDied","Data":"09c33f4a31f5bda34a21e934d0d87540be0a2d9da3478410832b7dca869242a3"} Jul 17 22:09:04.079967 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:04.079732 2566 scope.go:117] "RemoveContainer" containerID="a25b3d7edb8880dc519e237178f80a48e8dbd859ca9f84786999b515ab57e425" Jul 17 22:09:04.080106 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:04.080040 2566 scope.go:117] "RemoveContainer" containerID="09c33f4a31f5bda34a21e934d0d87540be0a2d9da3478410832b7dca869242a3" Jul 17 22:09:04.080337 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:09:04.080304 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl_odh-ai-gateway-infra(f4a45201-bf95-49f4-abbb-c951d614eeb1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" Jul 17 22:09:05.105452 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:05.105421 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl"] Jul 17 22:09:05.244327 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:05.244300 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" Jul 17 22:09:05.369284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:05.369148 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a45201-bf95-49f4-abbb-c951d614eeb1-maas-api-service-ca\") pod \"f4a45201-bf95-49f4-abbb-c951d614eeb1\" (UID: \"f4a45201-bf95-49f4-abbb-c951d614eeb1\") " Jul 17 22:09:05.369531 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:05.369508 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a45201-bf95-49f4-abbb-c951d614eeb1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4a45201-bf95-49f4-abbb-c951d614eeb1" (UID: "f4a45201-bf95-49f4-abbb-c951d614eeb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:09:05.469974 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:05.469941 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4a45201-bf95-49f4-abbb-c951d614eeb1-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:06.092757 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:06.092730 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" Jul 17 22:09:06.092949 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:06.092755 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl" event={"ID":"f4a45201-bf95-49f4-abbb-c951d614eeb1","Type":"ContainerDied","Data":"01ddacb830af64cb63aff2605e3257b172845198c43a97dc3cf3f642d0912a23"} Jul 17 22:09:06.092949 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:06.092811 2566 scope.go:117] "RemoveContainer" containerID="09c33f4a31f5bda34a21e934d0d87540be0a2d9da3478410832b7dca869242a3" Jul 17 22:09:06.118130 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:06.118101 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl"] Jul 17 22:09:06.123715 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:06.123691 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-wmmpl"] Jul 17 22:09:06.598360 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:06.598327 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:09:07.299436 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:07.299406 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" path="/var/lib/kubelet/pods/f4a45201-bf95-49f4-abbb-c951d614eeb1/volumes" Jul 17 22:09:09.890432 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890396 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s"] Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890748 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890759 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890767 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890773 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890782 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890789 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890799 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890804 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890823 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.890855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890829 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.891186 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890885 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.891186 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890892 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:09.891186 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890897 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:09.891186 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890903 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:09.891186 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.890909 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:09.895408 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.895389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" Jul 17 22:09:09.898296 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.898251 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:09:09.899592 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.899566 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:09:09.901945 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:09.901918 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s"] Jul 17 22:09:10.011129 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:10.011093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bae5da-2995-4844-8ec7-7de60e31de36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s\" (UID: \"63bae5da-2995-4844-8ec7-7de60e31de36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" Jul 17 22:09:10.112329 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:10.112289 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bae5da-2995-4844-8ec7-7de60e31de36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s\" (UID: \"63bae5da-2995-4844-8ec7-7de60e31de36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" Jul 17 22:09:10.112897 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:10.112878 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bae5da-2995-4844-8ec7-7de60e31de36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s\" (UID: \"63bae5da-2995-4844-8ec7-7de60e31de36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" Jul 17 22:09:10.207565 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:10.207527 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" Jul 17 22:09:10.363041 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:10.363014 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s"] Jul 17 22:09:10.365608 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:09:10.365581 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bae5da_2995_4844_8ec7_7de60e31de36.slice/crio-e2847b627c6f4036526a3bfe5ef0f63de244c8de5e63a071869a255347da481f WatchSource:0}: Error finding container e2847b627c6f4036526a3bfe5ef0f63de244c8de5e63a071869a255347da481f: Status 404 returned error can't find the container with id e2847b627c6f4036526a3bfe5ef0f63de244c8de5e63a071869a255347da481f Jul 17 22:09:11.118972 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.118942 2566 generic.go:356] "Generic (PLEG): container finished" podID="63bae5da-2995-4844-8ec7-7de60e31de36" containerID="c5778e73123304f86f4f2c3ef58990647406d91fbcb9d951fe140dab7c7e365a" exitCode=6 Jul 17 22:09:11.119364 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.119001 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" event={"ID":"63bae5da-2995-4844-8ec7-7de60e31de36","Type":"ContainerDied","Data":"c5778e73123304f86f4f2c3ef58990647406d91fbcb9d951fe140dab7c7e365a"} Jul 17 22:09:11.119364 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.119040 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" event={"ID":"63bae5da-2995-4844-8ec7-7de60e31de36","Type":"ContainerStarted","Data":"e2847b627c6f4036526a3bfe5ef0f63de244c8de5e63a071869a255347da481f"} Jul 17 22:09:11.119364 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.119221 2566 scope.go:117] "RemoveContainer" containerID="c5778e73123304f86f4f2c3ef58990647406d91fbcb9d951fe140dab7c7e365a" Jul 17 22:09:11.360739 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.360657 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s"] Jul 17 22:09:11.361141 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.361118 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:11.361141 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.361140 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a45201-bf95-49f4-abbb-c951d614eeb1" containerName="revoke-keys" Jul 17 22:09:11.361374 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.361255 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b53e1b1-109c-4b3e-9dd2-2321e6babb5f" containerName="revoke-keys" Jul 17 22:09:11.365657 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.365637 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.368330 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.368313 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8962d973-openshift-default-dockercfg-6klj9\"" Jul 17 22:09:11.377181 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.377159 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s"] Jul 17 22:09:11.423040 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423003 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-credential-socket\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423205 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-envoy\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423205 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423101 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-socket\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423205 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423133 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-podinfo\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423205 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423176 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-token\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423205 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423191 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istiod-ca-cert\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423447 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh49w\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-kube-api-access-vh49w\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423447 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423268 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-certs\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.423447 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.423305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-data\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524110 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524062 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-certs\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524110 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-data\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524142 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-credential-socket\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-envoy\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524201 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-socket\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-podinfo\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-token\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istiod-ca-cert\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524808 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524485 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh49w\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-kube-api-access-vh49w\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524808 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524545 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-certs\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524808 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524576 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-credential-socket\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524808 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-data\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.524808 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.524776 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-socket\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.525104 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.525087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istiod-ca-cert\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.526777 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.526758 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-envoy\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.526882 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.526801 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-podinfo\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.532807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.532784 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-token\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.532922 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.532837 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh49w\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-kube-api-access-vh49w\") pod \"e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.676870 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.676841 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:11.818926 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.818891 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s"] Jul 17 22:09:11.822783 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:09:11.822756 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e00e5e5_e2d7_462f_a1ae_9d70d3a906ad.slice/crio-2e0829911e0951b49c0ddea2fcc4f216ed1c8508098249f32c8b2b3e2c46a297 WatchSource:0}: Error finding container 2e0829911e0951b49c0ddea2fcc4f216ed1c8508098249f32c8b2b3e2c46a297: Status 404 returned error can't find the container with id 2e0829911e0951b49c0ddea2fcc4f216ed1c8508098249f32c8b2b3e2c46a297 Jul 17 22:09:11.824914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.824884 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:09:11.824977 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.824962 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:09:11.825010 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:11.824992 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:09:12.131975 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.131930 2566 generic.go:356] "Generic (PLEG): container finished" podID="63bae5da-2995-4844-8ec7-7de60e31de36" containerID="068415ec7a39a5798e8855963a51144ada5cdd8bc179db777c29f00228e59716" exitCode=6 Jul 17 22:09:12.132511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.132008 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" event={"ID":"63bae5da-2995-4844-8ec7-7de60e31de36","Type":"ContainerDied","Data":"068415ec7a39a5798e8855963a51144ada5cdd8bc179db777c29f00228e59716"} Jul 17 22:09:12.132511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.132390 2566 scope.go:117] "RemoveContainer" containerID="c5778e73123304f86f4f2c3ef58990647406d91fbcb9d951fe140dab7c7e365a" Jul 17 22:09:12.132736 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.132706 2566 scope.go:117] "RemoveContainer" containerID="068415ec7a39a5798e8855963a51144ada5cdd8bc179db777c29f00228e59716" Jul 17 22:09:12.132920 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:09:12.132903 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s_odh-ai-gateway-infra(63bae5da-2995-4844-8ec7-7de60e31de36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" Jul 17 22:09:12.134119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.134089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" event={"ID":"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad","Type":"ContainerStarted","Data":"876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb"} Jul 17 22:09:12.134204 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.134129 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" event={"ID":"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad","Type":"ContainerStarted","Data":"2e0829911e0951b49c0ddea2fcc4f216ed1c8508098249f32c8b2b3e2c46a297"} Jul 17 22:09:12.166958 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.166890 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" podStartSLOduration=1.166869986 podStartE2EDuration="1.166869986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:09:12.164339909 +0000 UTC m=+2489.434337639" watchObservedRunningTime="2026-07-17 22:09:12.166869986 +0000 UTC m=+2489.436867690" Jul 17 22:09:12.677787 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.677749 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:12.682658 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:12.682632 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:13.140762 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:13.140670 2566 scope.go:117] "RemoveContainer" containerID="068415ec7a39a5798e8855963a51144ada5cdd8bc179db777c29f00228e59716" Jul 17 22:09:13.141195 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:13.140804 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:13.141195 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:09:13.140904 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s_odh-ai-gateway-infra(63bae5da-2995-4844-8ec7-7de60e31de36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" Jul 17 22:09:13.141796 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:13.141782 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:16.905067 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:16.905031 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:09:25.208094 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:25.208059 2566 generic.go:356] "Generic (PLEG): container finished" podID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerID="876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb" exitCode=0 Jul 17 22:09:25.208554 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:25.208133 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" event={"ID":"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad","Type":"ContainerDied","Data":"876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb"} Jul 17 22:09:25.208599 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:25.208561 2566 scope.go:117] "RemoveContainer" containerID="876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb" Jul 17 22:09:25.209686 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:25.209655 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:09:25.209796 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:25.209725 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:09:25.209796 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:25.209775 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:09:26.216995 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:26.216964 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" event={"ID":"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad","Type":"ContainerStarted","Data":"27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0"} Jul 17 22:09:26.292722 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:26.292695 2566 scope.go:117] "RemoveContainer" containerID="068415ec7a39a5798e8855963a51144ada5cdd8bc179db777c29f00228e59716" Jul 17 22:09:26.677995 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:26.677956 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:26.679182 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:26.679159 2566 patch_prober.go:28] interesting pod/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 17 22:09:26.679267 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:26.679209 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 17 22:09:27.223490 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:27.223459 2566 generic.go:356] "Generic (PLEG): container finished" podID="63bae5da-2995-4844-8ec7-7de60e31de36" containerID="a91d4dd5a329d57740f41cefc85e6f3bf696a6302bbb808115b989dadaa9ee1a" exitCode=6 Jul 17 22:09:27.223918 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:27.223535 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" event={"ID":"63bae5da-2995-4844-8ec7-7de60e31de36","Type":"ContainerDied","Data":"a91d4dd5a329d57740f41cefc85e6f3bf696a6302bbb808115b989dadaa9ee1a"} Jul 17 22:09:27.223918 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:27.223592 2566 scope.go:117] "RemoveContainer" containerID="068415ec7a39a5798e8855963a51144ada5cdd8bc179db777c29f00228e59716" Jul 17 22:09:27.224050 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:27.224015 2566 scope.go:117] "RemoveContainer" containerID="a91d4dd5a329d57740f41cefc85e6f3bf696a6302bbb808115b989dadaa9ee1a" Jul 17 22:09:27.224255 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:09:27.224212 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s_odh-ai-gateway-infra(63bae5da-2995-4844-8ec7-7de60e31de36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" Jul 17 22:09:27.677810 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:27.677769 2566 patch_prober.go:28] interesting pod/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 17 22:09:27.678021 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:27.677835 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 17 22:09:28.141599 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.141510 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:28.248626 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.248589 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s"] Jul 17 22:09:28.431586 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.431559 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" Jul 17 22:09:28.481783 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.481754 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bae5da-2995-4844-8ec7-7de60e31de36-maas-api-service-ca\") pod \"63bae5da-2995-4844-8ec7-7de60e31de36\" (UID: \"63bae5da-2995-4844-8ec7-7de60e31de36\") " Jul 17 22:09:28.482082 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.482056 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bae5da-2995-4844-8ec7-7de60e31de36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63bae5da-2995-4844-8ec7-7de60e31de36" (UID: "63bae5da-2995-4844-8ec7-7de60e31de36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:09:28.582962 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.582920 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63bae5da-2995-4844-8ec7-7de60e31de36-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:28.678115 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.678068 2566 patch_prober.go:28] interesting pod/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 17 22:09:28.678341 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:28.678135 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 17 22:09:29.235374 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.235349 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" Jul 17 22:09:29.235557 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.235376 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s" event={"ID":"63bae5da-2995-4844-8ec7-7de60e31de36","Type":"ContainerDied","Data":"e2847b627c6f4036526a3bfe5ef0f63de244c8de5e63a071869a255347da481f"} Jul 17 22:09:29.235557 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.235430 2566 scope.go:117] "RemoveContainer" containerID="a91d4dd5a329d57740f41cefc85e6f3bf696a6302bbb808115b989dadaa9ee1a" Jul 17 22:09:29.259429 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.259364 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s"] Jul 17 22:09:29.265695 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.265666 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-qhv6s"] Jul 17 22:09:29.298177 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.298141 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" path="/var/lib/kubelet/pods/63bae5da-2995-4844-8ec7-7de60e31de36/volumes" Jul 17 22:09:29.677461 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.677425 2566 patch_prober.go:28] interesting pod/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 17 22:09:29.677614 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:29.677493 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 17 22:09:30.681960 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:30.681928 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:30.683061 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:30.683041 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:38.100142 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:38.100105 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:09:51.475515 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:51.475479 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s"] Jul 17 22:09:51.476088 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:51.475707 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" containerID="cri-o://27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0" gracePeriod=30 Jul 17 22:09:56.745774 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.745708 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:56.843448 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843412 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-socket\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843448 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843451 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-certs\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843472 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istiod-ca-cert\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843494 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh49w\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-kube-api-access-vh49w\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843532 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-token\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843573 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-podinfo\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843626 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-data\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843659 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-credential-socket\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.843704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843696 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-envoy\") pod \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\" (UID: \"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad\") " Jul 17 22:09:56.844060 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843812 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:09:56.844060 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843823 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:09:56.844060 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.843898 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:09:56.844060 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.844023 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:09:56.844285 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.844127 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.844285 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.844155 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.844285 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.844172 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.844285 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.844185 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.845073 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.845038 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:09:56.846067 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.846038 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:09:56.846067 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.846045 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-kube-api-access-vh49w" (OuterVolumeSpecName: "kube-api-access-vh49w") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "kube-api-access-vh49w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:09:56.846195 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.846063 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:09:56.846253 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.846215 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" (UID: "9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:09:56.945326 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.945289 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.945326 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.945322 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.945326 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.945331 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh49w\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-kube-api-access-vh49w\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.945563 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.945341 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:56.945563 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:56.945350 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:09:57.371277 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.371216 2566 generic.go:356] "Generic (PLEG): container finished" podID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerID="27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0" exitCode=0 Jul 17 22:09:57.371277 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.371277 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" event={"ID":"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad","Type":"ContainerDied","Data":"27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0"} Jul 17 22:09:57.371506 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.371292 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" Jul 17 22:09:57.371506 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.371322 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s" event={"ID":"9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad","Type":"ContainerDied","Data":"2e0829911e0951b49c0ddea2fcc4f216ed1c8508098249f32c8b2b3e2c46a297"} Jul 17 22:09:57.371506 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.371344 2566 scope.go:117] "RemoveContainer" containerID="27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0" Jul 17 22:09:57.380337 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.380312 2566 scope.go:117] "RemoveContainer" containerID="876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb" Jul 17 22:09:57.388984 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.388965 2566 scope.go:117] "RemoveContainer" containerID="27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0" Jul 17 22:09:57.389286 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:09:57.389268 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0\": container with ID starting with 27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0 not found: ID does not exist" containerID="27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0" Jul 17 22:09:57.389352 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.389293 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0"} err="failed to get container status \"27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0\": rpc error: code = NotFound desc = could not find container \"27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0\": container with ID starting with 27c9c560a0a983c38eb536a100e66075af4a7fff524579f056b5c310e226d3b0 not found: ID does not exist" Jul 17 22:09:57.389352 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.389311 2566 scope.go:117] "RemoveContainer" containerID="876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb" Jul 17 22:09:57.389549 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:09:57.389536 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb\": container with ID starting with 876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb not found: ID does not exist" containerID="876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb" Jul 17 22:09:57.389596 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.389553 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb"} err="failed to get container status \"876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb\": rpc error: code = NotFound desc = could not find container \"876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb\": container with ID starting with 876ae59b85b3095aab9653118e4ab631d80a5ff74b36198720ef412922572dcb not found: ID does not exist" Jul 17 22:09:57.389958 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.389940 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s"] Jul 17 22:09:57.393518 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:57.393493 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8962d973-openshift-default-67b6f5947f-b8l6s"] Jul 17 22:09:59.297976 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.297944 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" path="/var/lib/kubelet/pods/9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad/volumes" Jul 17 22:09:59.934059 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934019 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd"] Jul 17 22:09:59.934427 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934413 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:09:59.934477 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934429 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:09:59.934477 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934439 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" Jul 17 22:09:59.934477 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934445 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" Jul 17 22:09:59.934477 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934460 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" Jul 17 22:09:59.934477 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934465 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" Jul 17 22:09:59.934477 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934475 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:09:59.934655 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934481 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:09:59.934655 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934528 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:09:59.934655 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934537 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" Jul 17 22:09:59.934655 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934543 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:09:59.934655 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.934550 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e00e5e5-e2d7-462f-a1ae-9d70d3a906ad" containerName="istio-proxy" Jul 17 22:09:59.939245 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.939212 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" Jul 17 22:09:59.941960 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.941937 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:09:59.943121 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.943096 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:09:59.945366 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:09:59.945342 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd"] Jul 17 22:10:00.073022 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:00.072977 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60d430b-e833-462c-a93c-e355938c8281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd\" (UID: \"a60d430b-e833-462c-a93c-e355938c8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" Jul 17 22:10:00.173675 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:00.173632 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60d430b-e833-462c-a93c-e355938c8281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd\" (UID: \"a60d430b-e833-462c-a93c-e355938c8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" Jul 17 22:10:00.174288 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:00.174267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60d430b-e833-462c-a93c-e355938c8281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd\" (UID: \"a60d430b-e833-462c-a93c-e355938c8281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" Jul 17 22:10:00.252621 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:00.252531 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" Jul 17 22:10:00.386593 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:00.386570 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd"] Jul 17 22:10:00.389500 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:10:00.389472 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60d430b_e833_462c_a93c_e355938c8281.slice/crio-63dc915d24288374a3497a2dac6707a253f17f45fc231d0a38a514cf6b437f7a WatchSource:0}: Error finding container 63dc915d24288374a3497a2dac6707a253f17f45fc231d0a38a514cf6b437f7a: Status 404 returned error can't find the container with id 63dc915d24288374a3497a2dac6707a253f17f45fc231d0a38a514cf6b437f7a Jul 17 22:10:01.394876 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:01.394773 2566 generic.go:356] "Generic (PLEG): container finished" podID="a60d430b-e833-462c-a93c-e355938c8281" containerID="7a50343625e40d446fd384f4376a1d79857fb189016c07fd85e1a7a4bdf68d4d" exitCode=6 Jul 17 22:10:01.394876 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:01.394836 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" event={"ID":"a60d430b-e833-462c-a93c-e355938c8281","Type":"ContainerDied","Data":"7a50343625e40d446fd384f4376a1d79857fb189016c07fd85e1a7a4bdf68d4d"} Jul 17 22:10:01.395431 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:01.394883 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" event={"ID":"a60d430b-e833-462c-a93c-e355938c8281","Type":"ContainerStarted","Data":"63dc915d24288374a3497a2dac6707a253f17f45fc231d0a38a514cf6b437f7a"} Jul 17 22:10:01.395431 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:01.395082 2566 scope.go:117] "RemoveContainer" containerID="7a50343625e40d446fd384f4376a1d79857fb189016c07fd85e1a7a4bdf68d4d" Jul 17 22:10:02.401556 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:02.401453 2566 generic.go:356] "Generic (PLEG): container finished" podID="a60d430b-e833-462c-a93c-e355938c8281" containerID="275d840e64105669faa0e5400e8fe2ee6185f43d1eebe1c918ed88290d1fb567" exitCode=6 Jul 17 22:10:02.401556 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:02.401510 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" event={"ID":"a60d430b-e833-462c-a93c-e355938c8281","Type":"ContainerDied","Data":"275d840e64105669faa0e5400e8fe2ee6185f43d1eebe1c918ed88290d1fb567"} Jul 17 22:10:02.402124 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:02.401579 2566 scope.go:117] "RemoveContainer" containerID="7a50343625e40d446fd384f4376a1d79857fb189016c07fd85e1a7a4bdf68d4d" Jul 17 22:10:02.402124 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:02.401784 2566 scope.go:117] "RemoveContainer" containerID="275d840e64105669faa0e5400e8fe2ee6185f43d1eebe1c918ed88290d1fb567" Jul 17 22:10:02.402124 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:10:02.402055 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd_odh-ai-gateway-infra(a60d430b-e833-462c-a93c-e355938c8281)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" podUID="a60d430b-e833-462c-a93c-e355938c8281" Jul 17 22:10:03.408538 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:03.408515 2566 scope.go:117] "RemoveContainer" containerID="275d840e64105669faa0e5400e8fe2ee6185f43d1eebe1c918ed88290d1fb567" Jul 17 22:10:03.408921 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:10:03.408694 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd_odh-ai-gateway-infra(a60d430b-e833-462c-a93c-e355938c8281)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" podUID="a60d430b-e833-462c-a93c-e355938c8281" Jul 17 22:10:10.856914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.856876 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h"] Jul 17 22:10:10.857463 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.857441 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:10:10.857536 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.857466 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:10:10.857590 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.857569 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63bae5da-2995-4844-8ec7-7de60e31de36" containerName="revoke-keys" Jul 17 22:10:10.862338 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.862314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.866073 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.866052 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-539e6669-openshift-default-dockercfg-nw9c9\"" Jul 17 22:10:10.892100 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.892046 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h"] Jul 17 22:10:10.965549 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965498 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-envoy\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.965549 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9f4c29-5736-4397-80d5-55964faa0622-istiod-ca-cert\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.965811 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-socket\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.965811 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965623 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-credential-socket\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.965811 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-certs\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.965811 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965743 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9f4c29-5736-4397-80d5-55964faa0622-istio-podinfo\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.965811 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965784 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-data\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.966076 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965818 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6szm7\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-kube-api-access-6szm7\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:10.966076 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:10.965847 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-istio-token\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.066609 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.066570 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6szm7\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-kube-api-access-6szm7\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.066609 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.066613 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-istio-token\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.066827 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.066670 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-envoy\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.066827 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.066695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9f4c29-5736-4397-80d5-55964faa0622-istiod-ca-cert\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.066935 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.066903 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-socket\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.066973 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.066951 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-credential-socket\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067012 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.066979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-certs\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067086 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.067073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9f4c29-5736-4397-80d5-55964faa0622-istio-podinfo\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067167 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.067124 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-data\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067291 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.067267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-socket\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067365 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.067322 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-credential-socket\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067459 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.067431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-certs\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067583 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.067484 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-data\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.067583 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.067508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9f4c29-5736-4397-80d5-55964faa0622-istiod-ca-cert\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.069127 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.069104 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-envoy\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.069516 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.069497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9f4c29-5736-4397-80d5-55964faa0622-istio-podinfo\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.075406 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.075382 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-istio-token\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.075527 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.075509 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6szm7\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-kube-api-access-6szm7\") pod \"e2e-mt-539e6669-openshift-default-787884b675-xjb7h\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.173870 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.173837 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:11.330524 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:10:11.330488 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9f4c29_5736_4397_80d5_55964faa0622.slice/crio-046c23b218ead722010bf42c1e313be53281af3895d368618de82dfe5e632050 WatchSource:0}: Error finding container 046c23b218ead722010bf42c1e313be53281af3895d368618de82dfe5e632050: Status 404 returned error can't find the container with id 046c23b218ead722010bf42c1e313be53281af3895d368618de82dfe5e632050 Jul 17 22:10:11.331796 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.331765 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h"] Jul 17 22:10:11.332668 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.332637 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:10:11.332744 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.332703 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:10:11.332744 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.332732 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:10:11.449833 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.449717 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" event={"ID":"7c9f4c29-5736-4397-80d5-55964faa0622","Type":"ContainerStarted","Data":"1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8"} Jul 17 22:10:11.449833 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.449802 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" event={"ID":"7c9f4c29-5736-4397-80d5-55964faa0622","Type":"ContainerStarted","Data":"046c23b218ead722010bf42c1e313be53281af3895d368618de82dfe5e632050"} Jul 17 22:10:11.480933 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:11.480868 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" podStartSLOduration=1.480846522 podStartE2EDuration="1.480846522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:10:11.478996721 +0000 UTC m=+2548.748994426" watchObservedRunningTime="2026-07-17 22:10:11.480846522 +0000 UTC m=+2548.750844227" Jul 17 22:10:12.174183 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:12.174137 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:12.179107 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:12.179081 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:12.456356 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:12.456263 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:12.457365 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:12.457345 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:18.293146 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:18.293112 2566 scope.go:117] "RemoveContainer" containerID="275d840e64105669faa0e5400e8fe2ee6185f43d1eebe1c918ed88290d1fb567" Jul 17 22:10:19.211308 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:19.211275 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:10:19.494525 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:19.494440 2566 generic.go:356] "Generic (PLEG): container finished" podID="a60d430b-e833-462c-a93c-e355938c8281" containerID="7d68b92078ac2f4b995bcd9a0d696d8bc63d944418a955072a38f73606505da9" exitCode=6 Jul 17 22:10:19.494525 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:19.494503 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" event={"ID":"a60d430b-e833-462c-a93c-e355938c8281","Type":"ContainerDied","Data":"7d68b92078ac2f4b995bcd9a0d696d8bc63d944418a955072a38f73606505da9"} Jul 17 22:10:19.494984 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:19.494554 2566 scope.go:117] "RemoveContainer" containerID="275d840e64105669faa0e5400e8fe2ee6185f43d1eebe1c918ed88290d1fb567" Jul 17 22:10:19.494984 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:19.494943 2566 scope.go:117] "RemoveContainer" containerID="7d68b92078ac2f4b995bcd9a0d696d8bc63d944418a955072a38f73606505da9" Jul 17 22:10:19.495222 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:10:19.495189 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd_odh-ai-gateway-infra(a60d430b-e833-462c-a93c-e355938c8281)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" podUID="a60d430b-e833-462c-a93c-e355938c8281" Jul 17 22:10:20.520369 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:20.520333 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd"] Jul 17 22:10:20.718029 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:20.718006 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" Jul 17 22:10:20.775738 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:20.775634 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60d430b-e833-462c-a93c-e355938c8281-maas-api-service-ca\") pod \"a60d430b-e833-462c-a93c-e355938c8281\" (UID: \"a60d430b-e833-462c-a93c-e355938c8281\") " Jul 17 22:10:20.776644 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:20.776614 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60d430b-e833-462c-a93c-e355938c8281-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a60d430b-e833-462c-a93c-e355938c8281" (UID: "a60d430b-e833-462c-a93c-e355938c8281"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:10:20.877394 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:20.877341 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60d430b-e833-462c-a93c-e355938c8281-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:10:21.506385 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:21.506356 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" Jul 17 22:10:21.506573 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:21.506359 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd" event={"ID":"a60d430b-e833-462c-a93c-e355938c8281","Type":"ContainerDied","Data":"63dc915d24288374a3497a2dac6707a253f17f45fc231d0a38a514cf6b437f7a"} Jul 17 22:10:21.506573 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:21.506497 2566 scope.go:117] "RemoveContainer" containerID="7d68b92078ac2f4b995bcd9a0d696d8bc63d944418a955072a38f73606505da9" Jul 17 22:10:21.523269 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:21.523219 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd"] Jul 17 22:10:21.528766 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:21.528739 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-kk9xd"] Jul 17 22:10:23.298588 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:23.298558 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60d430b-e833-462c-a93c-e355938c8281" path="/var/lib/kubelet/pods/a60d430b-e833-462c-a93c-e355938c8281/volumes" Jul 17 22:10:27.458169 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.458123 2566 patch_prober.go:28] interesting pod/e2e-mt-539e6669-openshift-default-787884b675-xjb7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 17 22:10:27.458680 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.458210 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 17 22:10:27.540005 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.539973 2566 generic.go:356] "Generic (PLEG): container finished" podID="7c9f4c29-5736-4397-80d5-55964faa0622" containerID="1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8" exitCode=0 Jul 17 22:10:27.540166 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.540023 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" event={"ID":"7c9f4c29-5736-4397-80d5-55964faa0622","Type":"ContainerDied","Data":"1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8"} Jul 17 22:10:27.540426 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.540411 2566 scope.go:117] "RemoveContainer" containerID="1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8" Jul 17 22:10:27.541478 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.541445 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:10:27.541560 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.541534 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:10:27.541598 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:27.541577 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:10:28.548620 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:28.548579 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" event={"ID":"7c9f4c29-5736-4397-80d5-55964faa0622","Type":"ContainerStarted","Data":"6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f"} Jul 17 22:10:29.174053 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:29.174013 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:29.174349 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:29.174316 2566 patch_prober.go:28] interesting pod/e2e-mt-539e6669-openshift-default-787884b675-xjb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 17 22:10:29.174428 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:29.174393 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 17 22:10:30.174608 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:30.174565 2566 patch_prober.go:28] interesting pod/e2e-mt-539e6669-openshift-default-787884b675-xjb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 17 22:10:30.175004 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:30.174644 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 17 22:10:31.174580 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:31.174544 2566 patch_prober.go:28] interesting pod/e2e-mt-539e6669-openshift-default-787884b675-xjb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 17 22:10:31.174846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:31.174599 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 17 22:10:32.174924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:32.174879 2566 patch_prober.go:28] interesting pod/e2e-mt-539e6669-openshift-default-787884b675-xjb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 17 22:10:32.175325 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:32.174966 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 17 22:10:33.178880 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:33.178797 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:33.179344 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:33.179050 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:33.179799 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:33.179782 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:10:58.108617 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:10:58.108582 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:11:18.801508 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:11:18.801419 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:11:34.815476 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:11:34.815439 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:12:49.180995 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:49.180958 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h"] Jul 17 22:12:49.181438 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:49.181256 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" containerID="cri-o://6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f" gracePeriod=30 Jul 17 22:12:53.512446 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:53.512403 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:12:54.434185 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.434161 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:12:54.569409 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569308 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-socket\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.569409 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569343 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-data\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.569409 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569371 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-credential-socket\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.569409 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569397 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-certs\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569418 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-envoy\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569462 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-istio-token\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569481 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6szm7\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-kube-api-access-6szm7\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569500 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9f4c29-5736-4397-80d5-55964faa0622-istio-podinfo\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569653 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9f4c29-5736-4397-80d5-55964faa0622-istiod-ca-cert\") pod \"7c9f4c29-5736-4397-80d5-55964faa0622\" (UID: \"7c9f4c29-5736-4397-80d5-55964faa0622\") " Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569686 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569716 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.569765 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.570012 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.570030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.570033 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.570758 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.570050 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.570758 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.570070 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9f4c29-5736-4397-80d5-55964faa0622-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:12:54.570758 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.570122 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:12:54.571907 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.571879 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-kube-api-access-6szm7" (OuterVolumeSpecName: "kube-api-access-6szm7") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "kube-api-access-6szm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:12:54.572005 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.571913 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c9f4c29-5736-4397-80d5-55964faa0622-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:12:54.572420 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.572399 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:12:54.572478 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.572422 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c9f4c29-5736-4397-80d5-55964faa0622" (UID: "7c9f4c29-5736-4397-80d5-55964faa0622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:12:54.671112 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.671069 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.671112 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.671105 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.671112 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.671115 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6szm7\" (UniqueName: \"kubernetes.io/projected/7c9f4c29-5736-4397-80d5-55964faa0622-kube-api-access-6szm7\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.671391 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.671125 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9f4c29-5736-4397-80d5-55964faa0622-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.671391 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.671134 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9f4c29-5736-4397-80d5-55964faa0622-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:54.671391 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:54.671142 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9f4c29-5736-4397-80d5-55964faa0622-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:12:55.303843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.303808 2566 generic.go:356] "Generic (PLEG): container finished" podID="7c9f4c29-5736-4397-80d5-55964faa0622" containerID="6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f" exitCode=0 Jul 17 22:12:55.304046 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.303943 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" Jul 17 22:12:55.304389 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.304363 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" event={"ID":"7c9f4c29-5736-4397-80d5-55964faa0622","Type":"ContainerDied","Data":"6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f"} Jul 17 22:12:55.304486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.304408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h" event={"ID":"7c9f4c29-5736-4397-80d5-55964faa0622","Type":"ContainerDied","Data":"046c23b218ead722010bf42c1e313be53281af3895d368618de82dfe5e632050"} Jul 17 22:12:55.304486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.304431 2566 scope.go:117] "RemoveContainer" containerID="6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f" Jul 17 22:12:55.315304 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.315282 2566 scope.go:117] "RemoveContainer" containerID="1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8" Jul 17 22:12:55.323866 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.323846 2566 scope.go:117] "RemoveContainer" containerID="6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f" Jul 17 22:12:55.324120 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:12:55.324101 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f\": container with ID starting with 6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f not found: ID does not exist" containerID="6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f" Jul 17 22:12:55.324165 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.324131 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f"} err="failed to get container status \"6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f\": rpc error: code = NotFound desc = could not find container \"6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f\": container with ID starting with 6ee6d82224e8ab925e89815190c89a795ac9fc3f31674ae792682049f75da58f not found: ID does not exist" Jul 17 22:12:55.324165 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.324149 2566 scope.go:117] "RemoveContainer" containerID="1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8" Jul 17 22:12:55.324414 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:12:55.324397 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8\": container with ID starting with 1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8 not found: ID does not exist" containerID="1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8" Jul 17 22:12:55.324461 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.324421 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8"} err="failed to get container status \"1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8\": rpc error: code = NotFound desc = could not find container \"1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8\": container with ID starting with 1e3123a90530ddc006c0c5143aa09cd73128556efb904e3f38307af37b3804f8 not found: ID does not exist" Jul 17 22:12:55.332099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.332077 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h"] Jul 17 22:12:55.335709 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:55.335689 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-539e6669-openshift-default-787884b675-xjb7h"] Jul 17 22:12:57.298818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:12:57.298781 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" path="/var/lib/kubelet/pods/7c9f4c29-5736-4397-80d5-55964faa0622/volumes" Jul 17 22:13:08.105510 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:08.105473 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:13:15.791954 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.791914 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4"] Jul 17 22:13:15.792353 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792312 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" Jul 17 22:13:15.792353 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792326 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" Jul 17 22:13:15.792353 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792342 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:15.792353 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792347 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:15.792486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792364 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:15.792486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792369 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:15.792486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792421 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:15.792486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792428 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" Jul 17 22:13:15.792486 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.792437 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:15.794924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.794907 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" Jul 17 22:13:15.797512 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.797493 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:13:15.797584 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.797494 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:13:15.801374 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.801337 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4"] Jul 17 22:13:15.850703 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.850662 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34639d-c1e0-44a6-9127-0221ab2093d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4\" (UID: \"3d34639d-c1e0-44a6-9127-0221ab2093d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" Jul 17 22:13:15.951296 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.951264 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34639d-c1e0-44a6-9127-0221ab2093d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4\" (UID: \"3d34639d-c1e0-44a6-9127-0221ab2093d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" Jul 17 22:13:15.951935 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:15.951914 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34639d-c1e0-44a6-9127-0221ab2093d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4\" (UID: \"3d34639d-c1e0-44a6-9127-0221ab2093d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" Jul 17 22:13:16.106423 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:16.106331 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" Jul 17 22:13:16.242719 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:16.242696 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4"] Jul 17 22:13:16.244888 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:13:16.244865 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d34639d_c1e0_44a6_9127_0221ab2093d8.slice/crio-a2ffae365462219c830bf6df5d22863a57973ab9b0854ed4bb63d6791a1f9661 WatchSource:0}: Error finding container a2ffae365462219c830bf6df5d22863a57973ab9b0854ed4bb63d6791a1f9661: Status 404 returned error can't find the container with id a2ffae365462219c830bf6df5d22863a57973ab9b0854ed4bb63d6791a1f9661 Jul 17 22:13:16.411780 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:16.411749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" event={"ID":"3d34639d-c1e0-44a6-9127-0221ab2093d8","Type":"ContainerStarted","Data":"a2ffae365462219c830bf6df5d22863a57973ab9b0854ed4bb63d6791a1f9661"} Jul 17 22:13:17.418206 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:17.418173 2566 generic.go:356] "Generic (PLEG): container finished" podID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerID="c38142989e1e2204f514d5894a4efba45d2f61b1bb0198f7ac93049f5d63a723" exitCode=6 Jul 17 22:13:17.418949 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:17.418218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" event={"ID":"3d34639d-c1e0-44a6-9127-0221ab2093d8","Type":"ContainerDied","Data":"c38142989e1e2204f514d5894a4efba45d2f61b1bb0198f7ac93049f5d63a723"} Jul 17 22:13:17.418949 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:17.418443 2566 scope.go:117] "RemoveContainer" containerID="c38142989e1e2204f514d5894a4efba45d2f61b1bb0198f7ac93049f5d63a723" Jul 17 22:13:18.425647 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:18.425614 2566 generic.go:356] "Generic (PLEG): container finished" podID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerID="63316545d4b826d8feb8a16adf69fc3ba44d99ffe748399df45bef65b7cf0fe3" exitCode=6 Jul 17 22:13:18.426122 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:18.425689 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" event={"ID":"3d34639d-c1e0-44a6-9127-0221ab2093d8","Type":"ContainerDied","Data":"63316545d4b826d8feb8a16adf69fc3ba44d99ffe748399df45bef65b7cf0fe3"} Jul 17 22:13:18.426122 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:18.425739 2566 scope.go:117] "RemoveContainer" containerID="c38142989e1e2204f514d5894a4efba45d2f61b1bb0198f7ac93049f5d63a723" Jul 17 22:13:18.426122 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:18.425972 2566 scope.go:117] "RemoveContainer" containerID="63316545d4b826d8feb8a16adf69fc3ba44d99ffe748399df45bef65b7cf0fe3" Jul 17 22:13:18.426341 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:18.426297 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4_odh-ai-gateway-infra(3d34639d-c1e0-44a6-9127-0221ab2093d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" Jul 17 22:13:19.432445 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:19.432418 2566 scope.go:117] "RemoveContainer" containerID="63316545d4b826d8feb8a16adf69fc3ba44d99ffe748399df45bef65b7cf0fe3" Jul 17 22:13:19.432822 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:19.432681 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4_odh-ai-gateway-infra(3d34639d-c1e0-44a6-9127-0221ab2093d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" Jul 17 22:13:30.105744 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.105710 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd"] Jul 17 22:13:30.106141 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.106113 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" Jul 17 22:13:30.106141 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.106124 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" Jul 17 22:13:30.106141 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.106138 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:30.106278 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.106145 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:30.106278 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.106192 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9f4c29-5736-4397-80d5-55964faa0622" containerName="istio-proxy" Jul 17 22:13:30.106278 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.106203 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60d430b-e833-462c-a93c-e355938c8281" containerName="revoke-keys" Jul 17 22:13:30.109474 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.109451 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" Jul 17 22:13:30.115643 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.115613 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd"] Jul 17 22:13:30.174680 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.174639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e4cb88-4f7f-42c7-8374-dcbfb875c3da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd\" (UID: \"71e4cb88-4f7f-42c7-8374-dcbfb875c3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" Jul 17 22:13:30.275540 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.275504 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e4cb88-4f7f-42c7-8374-dcbfb875c3da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd\" (UID: \"71e4cb88-4f7f-42c7-8374-dcbfb875c3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" Jul 17 22:13:30.276196 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.276175 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e4cb88-4f7f-42c7-8374-dcbfb875c3da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd\" (UID: \"71e4cb88-4f7f-42c7-8374-dcbfb875c3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" Jul 17 22:13:30.292709 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.292682 2566 scope.go:117] "RemoveContainer" containerID="63316545d4b826d8feb8a16adf69fc3ba44d99ffe748399df45bef65b7cf0fe3" Jul 17 22:13:30.420839 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.420808 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" Jul 17 22:13:30.633439 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:30.633413 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd"] Jul 17 22:13:30.635406 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:13:30.635381 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e4cb88_4f7f_42c7_8374_dcbfb875c3da.slice/crio-f8a6a2b25af6d3d20822e7636dc74c1f74cabf75bc54fbd119245bf3ebdd89b5 WatchSource:0}: Error finding container f8a6a2b25af6d3d20822e7636dc74c1f74cabf75bc54fbd119245bf3ebdd89b5: Status 404 returned error can't find the container with id f8a6a2b25af6d3d20822e7636dc74c1f74cabf75bc54fbd119245bf3ebdd89b5 Jul 17 22:13:31.495482 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.495444 2566 generic.go:356] "Generic (PLEG): container finished" podID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerID="77b08892238d90e75875d1defbf369f286037794f982f13725d5a37ea0114765" exitCode=6 Jul 17 22:13:31.495936 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.495519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" event={"ID":"3d34639d-c1e0-44a6-9127-0221ab2093d8","Type":"ContainerDied","Data":"77b08892238d90e75875d1defbf369f286037794f982f13725d5a37ea0114765"} Jul 17 22:13:31.495936 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.495594 2566 scope.go:117] "RemoveContainer" containerID="63316545d4b826d8feb8a16adf69fc3ba44d99ffe748399df45bef65b7cf0fe3" Jul 17 22:13:31.496064 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.495938 2566 scope.go:117] "RemoveContainer" containerID="77b08892238d90e75875d1defbf369f286037794f982f13725d5a37ea0114765" Jul 17 22:13:31.496199 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:31.496178 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4_odh-ai-gateway-infra(3d34639d-c1e0-44a6-9127-0221ab2093d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" Jul 17 22:13:31.497192 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.497171 2566 generic.go:356] "Generic (PLEG): container finished" podID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerID="35573131eaf13af0aac05821dff186e1454b888af0f9b9d010fb5cea7e200e97" exitCode=6 Jul 17 22:13:31.497289 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.497220 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" event={"ID":"71e4cb88-4f7f-42c7-8374-dcbfb875c3da","Type":"ContainerDied","Data":"35573131eaf13af0aac05821dff186e1454b888af0f9b9d010fb5cea7e200e97"} Jul 17 22:13:31.497341 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.497293 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" event={"ID":"71e4cb88-4f7f-42c7-8374-dcbfb875c3da","Type":"ContainerStarted","Data":"f8a6a2b25af6d3d20822e7636dc74c1f74cabf75bc54fbd119245bf3ebdd89b5"} Jul 17 22:13:31.497494 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:31.497477 2566 scope.go:117] "RemoveContainer" containerID="35573131eaf13af0aac05821dff186e1454b888af0f9b9d010fb5cea7e200e97" Jul 17 22:13:32.506537 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.506505 2566 generic.go:356] "Generic (PLEG): container finished" podID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerID="92c0fc29834dfaa166aa3f8f71c8c9dbd6b07eba26165a6a3ab88372aa5efeac" exitCode=6 Jul 17 22:13:32.506993 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.506580 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" event={"ID":"71e4cb88-4f7f-42c7-8374-dcbfb875c3da","Type":"ContainerDied","Data":"92c0fc29834dfaa166aa3f8f71c8c9dbd6b07eba26165a6a3ab88372aa5efeac"} Jul 17 22:13:32.506993 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.506653 2566 scope.go:117] "RemoveContainer" containerID="35573131eaf13af0aac05821dff186e1454b888af0f9b9d010fb5cea7e200e97" Jul 17 22:13:32.506993 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.506885 2566 scope.go:117] "RemoveContainer" containerID="92c0fc29834dfaa166aa3f8f71c8c9dbd6b07eba26165a6a3ab88372aa5efeac" Jul 17 22:13:32.507136 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:32.507119 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd_odh-ai-gateway-infra(71e4cb88-4f7f-42c7-8374-dcbfb875c3da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" Jul 17 22:13:32.521336 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.521307 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4"] Jul 17 22:13:32.654784 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.654756 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" Jul 17 22:13:32.696337 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.696294 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34639d-c1e0-44a6-9127-0221ab2093d8-maas-api-service-ca\") pod \"3d34639d-c1e0-44a6-9127-0221ab2093d8\" (UID: \"3d34639d-c1e0-44a6-9127-0221ab2093d8\") " Jul 17 22:13:32.696623 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.696598 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d34639d-c1e0-44a6-9127-0221ab2093d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d34639d-c1e0-44a6-9127-0221ab2093d8" (UID: "3d34639d-c1e0-44a6-9127-0221ab2093d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:13:32.798049 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:32.797962 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34639d-c1e0-44a6-9127-0221ab2093d8-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:13:33.261362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.261332 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs"] Jul 17 22:13:33.261703 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.261690 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:33.261703 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.261703 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:33.261784 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.261716 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:33.261784 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.261722 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:33.261851 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.261804 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:33.261851 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.261814 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:33.264974 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.264959 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" Jul 17 22:13:33.271914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.271883 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs"] Jul 17 22:13:33.301869 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.301833 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d87bd6d-c180-43d7-9f96-47aa7565044f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs\" (UID: \"6d87bd6d-c180-43d7-9f96-47aa7565044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" Jul 17 22:13:33.403325 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.403290 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d87bd6d-c180-43d7-9f96-47aa7565044f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs\" (UID: \"6d87bd6d-c180-43d7-9f96-47aa7565044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" Jul 17 22:13:33.403898 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.403881 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d87bd6d-c180-43d7-9f96-47aa7565044f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs\" (UID: \"6d87bd6d-c180-43d7-9f96-47aa7565044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" Jul 17 22:13:33.512489 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.512407 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" Jul 17 22:13:33.512489 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.512402 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4" event={"ID":"3d34639d-c1e0-44a6-9127-0221ab2093d8","Type":"ContainerDied","Data":"a2ffae365462219c830bf6df5d22863a57973ab9b0854ed4bb63d6791a1f9661"} Jul 17 22:13:33.512993 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.512541 2566 scope.go:117] "RemoveContainer" containerID="77b08892238d90e75875d1defbf369f286037794f982f13725d5a37ea0114765" Jul 17 22:13:33.514578 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.514554 2566 scope.go:117] "RemoveContainer" containerID="92c0fc29834dfaa166aa3f8f71c8c9dbd6b07eba26165a6a3ab88372aa5efeac" Jul 17 22:13:33.514956 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:33.514924 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd_odh-ai-gateway-infra(71e4cb88-4f7f-42c7-8374-dcbfb875c3da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" Jul 17 22:13:33.540243 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.540204 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4"] Jul 17 22:13:33.543841 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.543818 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e3cf3d26-jhfb4"] Jul 17 22:13:33.576676 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.576651 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" Jul 17 22:13:33.917315 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:33.917293 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs"] Jul 17 22:13:33.921494 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:13:33.921462 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d87bd6d_c180_43d7_9f96_47aa7565044f.slice/crio-27e13de170f94984d89d67e2eb926d447fe3135e4705819284a286146b906030 WatchSource:0}: Error finding container 27e13de170f94984d89d67e2eb926d447fe3135e4705819284a286146b906030: Status 404 returned error can't find the container with id 27e13de170f94984d89d67e2eb926d447fe3135e4705819284a286146b906030 Jul 17 22:13:34.520700 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:34.520602 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" event={"ID":"6d87bd6d-c180-43d7-9f96-47aa7565044f","Type":"ContainerStarted","Data":"ce0f4cd2a027588e3e7abc2a7bc7e3763d69c1648cda5218d8b49dcfbaaea05f"} Jul 17 22:13:34.520700 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:34.520665 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" event={"ID":"6d87bd6d-c180-43d7-9f96-47aa7565044f","Type":"ContainerStarted","Data":"27e13de170f94984d89d67e2eb926d447fe3135e4705819284a286146b906030"} Jul 17 22:13:34.536691 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:34.536640 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" podStartSLOduration=1.5366227430000001 podStartE2EDuration="1.536622743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:34.534915422 +0000 UTC m=+2751.804913143" watchObservedRunningTime="2026-07-17 22:13:34.536622743 +0000 UTC m=+2751.806620447" Jul 17 22:13:35.304284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:35.304252 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" path="/var/lib/kubelet/pods/3d34639d-c1e0-44a6-9127-0221ab2093d8/volumes" Jul 17 22:13:35.527840 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:35.527800 2566 generic.go:356] "Generic (PLEG): container finished" podID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerID="ce0f4cd2a027588e3e7abc2a7bc7e3763d69c1648cda5218d8b49dcfbaaea05f" exitCode=6 Jul 17 22:13:35.528317 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:35.527876 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" event={"ID":"6d87bd6d-c180-43d7-9f96-47aa7565044f","Type":"ContainerDied","Data":"ce0f4cd2a027588e3e7abc2a7bc7e3763d69c1648cda5218d8b49dcfbaaea05f"} Jul 17 22:13:35.528317 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:35.528065 2566 scope.go:117] "RemoveContainer" containerID="ce0f4cd2a027588e3e7abc2a7bc7e3763d69c1648cda5218d8b49dcfbaaea05f" Jul 17 22:13:36.535346 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:36.535307 2566 generic.go:356] "Generic (PLEG): container finished" podID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerID="f2dc96a399ae96a79bd3d6a6fd497d209fa6b531fdb7d0938eaa7f7c89b108d9" exitCode=6 Jul 17 22:13:36.535786 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:36.535400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" event={"ID":"6d87bd6d-c180-43d7-9f96-47aa7565044f","Type":"ContainerDied","Data":"f2dc96a399ae96a79bd3d6a6fd497d209fa6b531fdb7d0938eaa7f7c89b108d9"} Jul 17 22:13:36.535786 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:36.535457 2566 scope.go:117] "RemoveContainer" containerID="ce0f4cd2a027588e3e7abc2a7bc7e3763d69c1648cda5218d8b49dcfbaaea05f" Jul 17 22:13:36.535786 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:36.535579 2566 scope.go:117] "RemoveContainer" containerID="f2dc96a399ae96a79bd3d6a6fd497d209fa6b531fdb7d0938eaa7f7c89b108d9" Jul 17 22:13:36.535786 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:36.535774 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs_odh-ai-gateway-infra(6d87bd6d-c180-43d7-9f96-47aa7565044f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" Jul 17 22:13:37.541997 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:37.541972 2566 scope.go:117] "RemoveContainer" containerID="f2dc96a399ae96a79bd3d6a6fd497d209fa6b531fdb7d0938eaa7f7c89b108d9" Jul 17 22:13:37.542400 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:37.542152 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs_odh-ai-gateway-infra(6d87bd6d-c180-43d7-9f96-47aa7565044f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" Jul 17 22:13:44.292868 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.292839 2566 scope.go:117] "RemoveContainer" containerID="92c0fc29834dfaa166aa3f8f71c8c9dbd6b07eba26165a6a3ab88372aa5efeac" Jul 17 22:13:44.362586 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.362550 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm"] Jul 17 22:13:44.363171 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.363134 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:44.363171 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.363162 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:44.363380 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.363288 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d34639d-c1e0-44a6-9127-0221ab2093d8" containerName="revoke-keys" Jul 17 22:13:44.367964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.367940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.370512 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.370492 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-be8c97-openshift-default-dockercfg-dj4qg\"" Jul 17 22:13:44.377999 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.377970 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm"] Jul 17 22:13:44.505634 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-credential-socket\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.505634 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505641 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-data\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.505870 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505723 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc260277-90ab-435e-9348-1d0dda933471-istio-podinfo\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.505870 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505814 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-envoy\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.505870 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-socket\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.505971 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505880 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxbh\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-kube-api-access-qsxbh\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.505971 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505904 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc260277-90ab-435e-9348-1d0dda933471-istiod-ca-cert\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.505971 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505928 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-certs\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.506067 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.505968 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-istio-token\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.576493 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.576466 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" event={"ID":"71e4cb88-4f7f-42c7-8374-dcbfb875c3da","Type":"ContainerStarted","Data":"ae2d38afb779e48139dc828e215239c857fd283ea1d160737d2b2b48d10fe232"} Jul 17 22:13:44.593630 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.593579 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" podStartSLOduration=14.593562044 podStartE2EDuration="14.593562044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:44.591846845 +0000 UTC m=+2761.861844570" watchObservedRunningTime="2026-07-17 22:13:44.593562044 +0000 UTC m=+2761.863559747" Jul 17 22:13:44.606723 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606699 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-socket\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.606857 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606728 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxbh\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-kube-api-access-qsxbh\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.606857 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606751 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc260277-90ab-435e-9348-1d0dda933471-istiod-ca-cert\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.606857 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606774 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-certs\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.606857 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-istio-token\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.606857 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606844 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-credential-socket\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606863 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-data\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606894 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc260277-90ab-435e-9348-1d0dda933471-istio-podinfo\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.606965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-envoy\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607290 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.607141 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-socket\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607290 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.607181 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-certs\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607406 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.607294 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-data\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607462 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.607397 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-credential-socket\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.607658 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.607641 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc260277-90ab-435e-9348-1d0dda933471-istiod-ca-cert\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.609307 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.609280 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-envoy\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.609523 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.609507 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc260277-90ab-435e-9348-1d0dda933471-istio-podinfo\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.614925 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.614901 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-istio-token\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.615041 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.615025 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxbh\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-kube-api-access-qsxbh\") pod \"e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.680612 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.680573 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:44.824833 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.824807 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm"] Jul 17 22:13:44.826715 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:13:44.826694 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc260277_90ab_435e_9348_1d0dda933471.slice/crio-6c63a96377252733088b117ebc106fe68dd9364de787b6d6a16e0aad0b60397a WatchSource:0}: Error finding container 6c63a96377252733088b117ebc106fe68dd9364de787b6d6a16e0aad0b60397a: Status 404 returned error can't find the container with id 6c63a96377252733088b117ebc106fe68dd9364de787b6d6a16e0aad0b60397a Jul 17 22:13:44.828905 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.828873 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:13:44.828965 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.828941 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:13:44.829009 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:44.828968 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:13:45.583094 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.582995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" event={"ID":"bc260277-90ab-435e-9348-1d0dda933471","Type":"ContainerStarted","Data":"65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a"} Jul 17 22:13:45.583094 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.583070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" event={"ID":"bc260277-90ab-435e-9348-1d0dda933471","Type":"ContainerStarted","Data":"6c63a96377252733088b117ebc106fe68dd9364de787b6d6a16e0aad0b60397a"} Jul 17 22:13:45.584855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.584829 2566 generic.go:356] "Generic (PLEG): container finished" podID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerID="ae2d38afb779e48139dc828e215239c857fd283ea1d160737d2b2b48d10fe232" exitCode=6 Jul 17 22:13:45.584963 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.584867 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" event={"ID":"71e4cb88-4f7f-42c7-8374-dcbfb875c3da","Type":"ContainerDied","Data":"ae2d38afb779e48139dc828e215239c857fd283ea1d160737d2b2b48d10fe232"} Jul 17 22:13:45.584963 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.584921 2566 scope.go:117] "RemoveContainer" containerID="92c0fc29834dfaa166aa3f8f71c8c9dbd6b07eba26165a6a3ab88372aa5efeac" Jul 17 22:13:45.585206 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.585188 2566 scope.go:117] "RemoveContainer" containerID="ae2d38afb779e48139dc828e215239c857fd283ea1d160737d2b2b48d10fe232" Jul 17 22:13:45.585476 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:45.585456 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd_odh-ai-gateway-infra(71e4cb88-4f7f-42c7-8374-dcbfb875c3da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" Jul 17 22:13:45.601559 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.601534 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd"] Jul 17 22:13:45.605639 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.605592 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" podStartSLOduration=1.605577273 podStartE2EDuration="1.605577273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:45.604377847 +0000 UTC m=+2762.874375572" watchObservedRunningTime="2026-07-17 22:13:45.605577273 +0000 UTC m=+2762.875574971" Jul 17 22:13:45.681766 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.681722 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:45.687209 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:45.687185 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:46.592921 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:46.592461 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:46.593509 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:46.593484 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:13:46.751374 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:46.751343 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" Jul 17 22:13:46.828387 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:46.828353 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e4cb88-4f7f-42c7-8374-dcbfb875c3da-maas-api-service-ca\") pod \"71e4cb88-4f7f-42c7-8374-dcbfb875c3da\" (UID: \"71e4cb88-4f7f-42c7-8374-dcbfb875c3da\") " Jul 17 22:13:46.828716 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:46.828694 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e4cb88-4f7f-42c7-8374-dcbfb875c3da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71e4cb88-4f7f-42c7-8374-dcbfb875c3da" (UID: "71e4cb88-4f7f-42c7-8374-dcbfb875c3da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:13:46.929391 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:46.929361 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e4cb88-4f7f-42c7-8374-dcbfb875c3da-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:13:47.598955 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:47.598858 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" event={"ID":"71e4cb88-4f7f-42c7-8374-dcbfb875c3da","Type":"ContainerDied","Data":"f8a6a2b25af6d3d20822e7636dc74c1f74cabf75bc54fbd119245bf3ebdd89b5"} Jul 17 22:13:47.598955 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:47.598941 2566 scope.go:117] "RemoveContainer" containerID="ae2d38afb779e48139dc828e215239c857fd283ea1d160737d2b2b48d10fe232" Jul 17 22:13:47.599494 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:47.598874 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd" Jul 17 22:13:47.627439 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:47.627408 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd"] Jul 17 22:13:47.634429 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:47.634402 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-5m2dd"] Jul 17 22:13:49.292867 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:49.292837 2566 scope.go:117] "RemoveContainer" containerID="f2dc96a399ae96a79bd3d6a6fd497d209fa6b531fdb7d0938eaa7f7c89b108d9" Jul 17 22:13:49.298708 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:49.298679 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" path="/var/lib/kubelet/pods/71e4cb88-4f7f-42c7-8374-dcbfb875c3da/volumes" Jul 17 22:13:49.610822 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:49.610730 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" event={"ID":"6d87bd6d-c180-43d7-9f96-47aa7565044f","Type":"ContainerStarted","Data":"0f3f4571561ce5b5124d7981b0ea564003e30da573b66c2113a5b297117795a6"} Jul 17 22:13:50.617924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:50.617886 2566 generic.go:356] "Generic (PLEG): container finished" podID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerID="0f3f4571561ce5b5124d7981b0ea564003e30da573b66c2113a5b297117795a6" exitCode=6 Jul 17 22:13:50.618417 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:50.617966 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" event={"ID":"6d87bd6d-c180-43d7-9f96-47aa7565044f","Type":"ContainerDied","Data":"0f3f4571561ce5b5124d7981b0ea564003e30da573b66c2113a5b297117795a6"} Jul 17 22:13:50.618417 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:50.618044 2566 scope.go:117] "RemoveContainer" containerID="f2dc96a399ae96a79bd3d6a6fd497d209fa6b531fdb7d0938eaa7f7c89b108d9" Jul 17 22:13:50.618417 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:50.618354 2566 scope.go:117] "RemoveContainer" containerID="0f3f4571561ce5b5124d7981b0ea564003e30da573b66c2113a5b297117795a6" Jul 17 22:13:50.618617 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:13:50.618594 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs_odh-ai-gateway-infra(6d87bd6d-c180-43d7-9f96-47aa7565044f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" Jul 17 22:13:50.638997 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:50.638964 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs"] Jul 17 22:13:51.773036 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:51.773010 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" Jul 17 22:13:51.876940 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:51.876901 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d87bd6d-c180-43d7-9f96-47aa7565044f-maas-api-service-ca\") pod \"6d87bd6d-c180-43d7-9f96-47aa7565044f\" (UID: \"6d87bd6d-c180-43d7-9f96-47aa7565044f\") " Jul 17 22:13:51.877310 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:51.877287 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d87bd6d-c180-43d7-9f96-47aa7565044f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d87bd6d-c180-43d7-9f96-47aa7565044f" (UID: "6d87bd6d-c180-43d7-9f96-47aa7565044f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:13:51.978270 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:51.978218 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d87bd6d-c180-43d7-9f96-47aa7565044f-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:13:52.636445 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.636420 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" Jul 17 22:13:52.636592 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.636418 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs" event={"ID":"6d87bd6d-c180-43d7-9f96-47aa7565044f","Type":"ContainerDied","Data":"27e13de170f94984d89d67e2eb926d447fe3135e4705819284a286146b906030"} Jul 17 22:13:52.636592 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.636550 2566 scope.go:117] "RemoveContainer" containerID="0f3f4571561ce5b5124d7981b0ea564003e30da573b66c2113a5b297117795a6" Jul 17 22:13:52.666044 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.665993 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs"] Jul 17 22:13:52.672519 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.672473 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-6nlrs"] Jul 17 22:13:52.676979 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.676956 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c"] Jul 17 22:13:52.677344 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677332 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:13:52.677386 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677346 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:13:52.677386 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677357 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:13:52.677386 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677363 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:13:52.677386 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677372 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:13:52.677386 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677378 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:13:52.677386 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677383 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:13:52.677386 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677388 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:13:52.677633 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677449 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:13:52.677633 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677458 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:13:52.677633 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677464 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:13:52.677633 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677471 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:13:52.677633 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.677475 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:13:52.681502 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.681482 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.684411 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.684393 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b09be9-openshift-default-dockercfg-xt82h\"" Jul 17 22:13:52.696138 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.696111 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c"] Jul 17 22:13:52.787580 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787546 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istiod-ca-cert\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.787580 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787585 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-data\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.788092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6nw\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-kube-api-access-4h6nw\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.788092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-certs\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.788092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787776 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-credential-socket\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.788092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-envoy\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.788092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787833 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-token\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.788092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787876 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-socket\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.788092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.787908 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-podinfo\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.888948 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.888857 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-certs\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.888943 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-credential-socket\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.888987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-envoy\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-token\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-socket\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889119 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889092 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-podinfo\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889400 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889184 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istiod-ca-cert\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889400 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889219 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-data\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889400 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6nw\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-kube-api-access-4h6nw\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889400 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889328 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-certs\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889400 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889335 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-credential-socket\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.889631 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.889580 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-socket\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.890088 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.890063 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istiod-ca-cert\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.890403 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.890381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-data\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.892401 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.892381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-envoy\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.892627 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.892606 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-podinfo\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.898590 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.898563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-token\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.898711 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.898695 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6nw\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-kube-api-access-4h6nw\") pod \"e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:52.992901 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:52.992865 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:53.141292 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.141267 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c"] Jul 17 22:13:53.143960 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:13:53.143918 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fcb6f1_3f5d_4667_93ed_141f67ea4853.slice/crio-587760d8849cac016fcce52bc58454fab4094cf38d9297006632aa52116ffd5d WatchSource:0}: Error finding container 587760d8849cac016fcce52bc58454fab4094cf38d9297006632aa52116ffd5d: Status 404 returned error can't find the container with id 587760d8849cac016fcce52bc58454fab4094cf38d9297006632aa52116ffd5d Jul 17 22:13:53.146464 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.146413 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:13:53.146563 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.146517 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:13:53.146605 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.146565 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:13:53.299750 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.299709 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" path="/var/lib/kubelet/pods/6d87bd6d-c180-43d7-9f96-47aa7565044f/volumes" Jul 17 22:13:53.643182 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.643148 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" event={"ID":"c0fcb6f1-3f5d-4667-93ed-141f67ea4853","Type":"ContainerStarted","Data":"5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b"} Jul 17 22:13:53.643377 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.643210 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" event={"ID":"c0fcb6f1-3f5d-4667-93ed-141f67ea4853","Type":"ContainerStarted","Data":"587760d8849cac016fcce52bc58454fab4094cf38d9297006632aa52116ffd5d"} Jul 17 22:13:53.664305 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.664257 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podStartSLOduration=1.664242387 podStartE2EDuration="1.664242387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:53.663013576 +0000 UTC m=+2770.933011279" watchObservedRunningTime="2026-07-17 22:13:53.664242387 +0000 UTC m=+2770.934240085" Jul 17 22:13:53.993215 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:53.993164 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:54.998739 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:54.998711 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:55.302908 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:55.302823 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:13:55.656127 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:55.656041 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:13:55.657373 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:13:55.657346 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:14:02.634417 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:02.634370 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 22:14:02.634877 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:02.634786 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 22:14:04.714239 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:04.714205 2566 generic.go:356] "Generic (PLEG): container finished" podID="bc260277-90ab-435e-9348-1d0dda933471" containerID="65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a" exitCode=0 Jul 17 22:14:04.714654 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:04.714285 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" event={"ID":"bc260277-90ab-435e-9348-1d0dda933471","Type":"ContainerDied","Data":"65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a"} Jul 17 22:14:04.714697 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:04.714690 2566 scope.go:117] "RemoveContainer" containerID="65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a" Jul 17 22:14:04.715787 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:04.715758 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:14:04.715849 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:04.715835 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:14:04.715888 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:04.715870 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:14:05.723261 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:05.723216 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" event={"ID":"bc260277-90ab-435e-9348-1d0dda933471","Type":"ContainerStarted","Data":"063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a"} Jul 17 22:14:06.682398 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:06.681908 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:14:06.682398 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:06.682040 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 22:14:06.682398 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:06.682086 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 22:14:07.681457 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:07.681416 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 22:14:07.682112 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:07.682077 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 22:14:08.681945 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:08.681893 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 22:14:08.682505 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:08.681987 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 22:14:09.684581 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:09.684541 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:14:09.684976 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:09.684848 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:14:09.685732 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:09.685707 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:14:10.110997 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:10.110892 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:14:11.698360 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:11.698326 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 22:14:11.698769 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:11.698378 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 22:14:12.765414 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:12.765376 2566 generic.go:356] "Generic (PLEG): container finished" podID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerID="5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b" exitCode=0 Jul 17 22:14:12.765943 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:12.765446 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" event={"ID":"c0fcb6f1-3f5d-4667-93ed-141f67ea4853","Type":"ContainerDied","Data":"5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b"} Jul 17 22:14:12.766012 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:12.765982 2566 scope.go:117] "RemoveContainer" containerID="5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b" Jul 17 22:14:12.767060 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:12.767024 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:14:12.767177 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:12.767112 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:14:12.767177 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:12.767165 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:14:13.787062 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:13.787020 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" event={"ID":"c0fcb6f1-3f5d-4667-93ed-141f67ea4853","Type":"ContainerStarted","Data":"252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5"} Jul 17 22:14:13.993084 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:13.993047 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:14:13.993320 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:13.993297 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 22:14:13.993374 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:13.993346 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 22:14:14.993895 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:14.993858 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 22:14:14.994401 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:14.993927 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 22:14:15.993568 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:15.993527 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 22:14:15.993789 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:15.993610 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 22:14:16.993779 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:16.993745 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 17 22:14:16.994136 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:16.993801 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 17 22:14:17.998083 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:17.998051 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:14:17.998472 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:17.998439 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:14:17.999327 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:17.999295 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:14:23.706167 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:23.706136 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:14:33.007484 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:33.007444 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:14:47.212655 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:14:47.212620 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:15:00.410581 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:00.410538 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:15:11.098839 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:11.098802 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:15:20.318154 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:20.318120 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm"] Jul 17 22:15:20.318619 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:20.318350 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" containerID="cri-o://063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a" gracePeriod=30 Jul 17 22:15:24.687325 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:24.687281 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 22:15:24.687702 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:24.687361 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 22:15:25.578277 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.578250 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:15:25.771877 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.771837 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-certs\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.771903 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-data\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.771927 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-istio-token\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.771961 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc260277-90ab-435e-9348-1d0dda933471-istio-podinfo\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772011 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-credential-socket\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772029 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-envoy\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772072 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-socket\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772101 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsxbh\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-kube-api-access-qsxbh\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772125 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc260277-90ab-435e-9348-1d0dda933471-istiod-ca-cert\") pod \"bc260277-90ab-435e-9348-1d0dda933471\" (UID: \"bc260277-90ab-435e-9348-1d0dda933471\") " Jul 17 22:15:25.772362 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772179 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:25.772843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772402 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:25.772843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772411 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:25.772843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772466 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.772843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772709 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc260277-90ab-435e-9348-1d0dda933471-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:15:25.772843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.772800 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:25.774425 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.774403 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc260277-90ab-435e-9348-1d0dda933471-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:15:25.774708 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.774687 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:25.774870 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.774854 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:15:25.774922 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.774892 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-kube-api-access-qsxbh" (OuterVolumeSpecName: "kube-api-access-qsxbh") pod "bc260277-90ab-435e-9348-1d0dda933471" (UID: "bc260277-90ab-435e-9348-1d0dda933471"). InnerVolumeSpecName "kube-api-access-qsxbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:15:25.873882 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873843 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxbh\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-kube-api-access-qsxbh\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.873882 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873874 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc260277-90ab-435e-9348-1d0dda933471-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.873882 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873884 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.873882 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873892 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc260277-90ab-435e-9348-1d0dda933471-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.874145 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873900 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc260277-90ab-435e-9348-1d0dda933471-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.874145 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873907 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.874145 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873917 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:25.874145 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:25.873926 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc260277-90ab-435e-9348-1d0dda933471-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:26.146763 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.146666 2566 generic.go:356] "Generic (PLEG): container finished" podID="bc260277-90ab-435e-9348-1d0dda933471" containerID="063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a" exitCode=0 Jul 17 22:15:26.146763 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.146719 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" Jul 17 22:15:26.146763 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.146754 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" event={"ID":"bc260277-90ab-435e-9348-1d0dda933471","Type":"ContainerDied","Data":"063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a"} Jul 17 22:15:26.147023 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.146818 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm" event={"ID":"bc260277-90ab-435e-9348-1d0dda933471","Type":"ContainerDied","Data":"6c63a96377252733088b117ebc106fe68dd9364de787b6d6a16e0aad0b60397a"} Jul 17 22:15:26.147023 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.146844 2566 scope.go:117] "RemoveContainer" containerID="063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a" Jul 17 22:15:26.156447 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.156430 2566 scope.go:117] "RemoveContainer" containerID="65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a" Jul 17 22:15:26.164628 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.164608 2566 scope.go:117] "RemoveContainer" containerID="063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a" Jul 17 22:15:26.164875 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:15:26.164856 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a\": container with ID starting with 063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a not found: ID does not exist" containerID="063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a" Jul 17 22:15:26.164924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.164884 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a"} err="failed to get container status \"063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a\": rpc error: code = NotFound desc = could not find container \"063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a\": container with ID starting with 063fcfa7250880b7f9a39086581fd28be02509c12c9440a284d1ad626524f49a not found: ID does not exist" Jul 17 22:15:26.164924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.164900 2566 scope.go:117] "RemoveContainer" containerID="65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a" Jul 17 22:15:26.165100 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:15:26.165085 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a\": container with ID starting with 65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a not found: ID does not exist" containerID="65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a" Jul 17 22:15:26.165143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.165104 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a"} err="failed to get container status \"65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a\": rpc error: code = NotFound desc = could not find container \"65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a\": container with ID starting with 65bc5fd6ea1fab6585e0c8247352419e90d9e1aa2e9037930345965e0d14fb4a not found: ID does not exist" Jul 17 22:15:26.168755 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.168729 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm"] Jul 17 22:15:26.172684 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:26.172662 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-be8c97-openshift-default-6f56565fd5-7jfdm"] Jul 17 22:15:27.298860 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:27.298826 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc260277-90ab-435e-9348-1d0dda933471" path="/var/lib/kubelet/pods/bc260277-90ab-435e-9348-1d0dda933471/volumes" Jul 17 22:15:34.302099 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:34.302019 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:15:46.464562 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:46.464519 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c"] Jul 17 22:15:46.465065 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:46.464778 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" containerID="cri-o://252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5" gracePeriod=30 Jul 17 22:15:47.360702 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.360646 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z"] Jul 17 22:15:47.361143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361113 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" Jul 17 22:15:47.361143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361133 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" Jul 17 22:15:47.361302 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361148 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:15:47.361302 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361157 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:15:47.361302 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361167 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:15:47.361302 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361196 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e4cb88-4f7f-42c7-8374-dcbfb875c3da" containerName="revoke-keys" Jul 17 22:15:47.361302 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361204 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" Jul 17 22:15:47.361302 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361210 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" Jul 17 22:15:47.361496 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361310 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" Jul 17 22:15:47.361496 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361323 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d87bd6d-c180-43d7-9f96-47aa7565044f" containerName="revoke-keys" Jul 17 22:15:47.361496 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.361331 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc260277-90ab-435e-9348-1d0dda933471" containerName="istio-proxy" Jul 17 22:15:47.366100 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.366077 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.368881 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.368858 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-95299c-openshift-default-dockercfg-gpc5m\"" Jul 17 22:15:47.375336 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.375310 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z"] Jul 17 22:15:47.458908 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.458850 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-podinfo\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459264 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-token\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459411 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459389 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-credential-socket\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459487 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istiod-ca-cert\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459487 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459461 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7v5k\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-kube-api-access-t7v5k\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459632 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459526 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-data\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459632 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459590 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-socket\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459632 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459614 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-certs\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.459815 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.459636 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-envoy\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.560623 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.560581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-socket\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.560623 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.560617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-certs\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.560637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-envoy\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.560666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-podinfo\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.560690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-token\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.560898 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-credential-socket\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istiod-ca-cert\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561054 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-socket\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561075 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7v5k\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-kube-api-access-t7v5k\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561188 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-certs\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561219 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-credential-socket\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561260 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-data\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-data\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.561818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.561628 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istiod-ca-cert\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.563221 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.563191 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-envoy\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.563553 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.563533 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-podinfo\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.569169 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.569143 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-token\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.569393 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.569374 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7v5k\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-kube-api-access-t7v5k\") pod \"e2e-api-a-95299c-openshift-default-6999645f65-2dp6z\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.678957 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.678919 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:47.825831 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.825804 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z"] Jul 17 22:15:47.828223 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:15:47.828196 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bc2a07_f612_4312_89ab_0bc77abbbd36.slice/crio-728b3faecc8dcb4ae0cfed54245a758d1cd07aae5f27a28addd4adc78fc347c7 WatchSource:0}: Error finding container 728b3faecc8dcb4ae0cfed54245a758d1cd07aae5f27a28addd4adc78fc347c7: Status 404 returned error can't find the container with id 728b3faecc8dcb4ae0cfed54245a758d1cd07aae5f27a28addd4adc78fc347c7 Jul 17 22:15:47.830403 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.830366 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:15:47.830489 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.830431 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:15:47.830489 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:47.830462 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:15:48.000472 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:48.000381 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 22:15:48.000472 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:48.000450 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 22:15:48.257159 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:48.257059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" event={"ID":"d7bc2a07-f612-4312-89ab-0bc77abbbd36","Type":"ContainerStarted","Data":"35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d"} Jul 17 22:15:48.257159 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:48.257127 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" event={"ID":"d7bc2a07-f612-4312-89ab-0bc77abbbd36","Type":"ContainerStarted","Data":"728b3faecc8dcb4ae0cfed54245a758d1cd07aae5f27a28addd4adc78fc347c7"} Jul 17 22:15:48.679626 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:48.679597 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:48.685039 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:48.685015 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:48.704668 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:48.704624 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podStartSLOduration=1.704609622 podStartE2EDuration="1.704609622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:48.279992924 +0000 UTC m=+2885.549990629" watchObservedRunningTime="2026-07-17 22:15:48.704609622 +0000 UTC m=+2885.974607307" Jul 17 22:15:49.262542 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:49.262508 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:49.263605 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:49.263581 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:15:51.722024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.721998 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:15:51.803268 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803157 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-envoy\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803268 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803198 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h6nw\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-kube-api-access-4h6nw\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803268 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803215 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-socket\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803268 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803265 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-token\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803604 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803321 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-podinfo\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803604 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803351 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-credential-socket\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803604 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803375 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-certs\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803604 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803424 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-data\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803604 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803455 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istiod-ca-cert\") pod \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\" (UID: \"c0fcb6f1-3f5d-4667-93ed-141f67ea4853\") " Jul 17 22:15:51.803604 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803569 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:51.803924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803669 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:51.803924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803827 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.803924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803847 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.803924 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.803905 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:15:51.804137 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.804036 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:51.804302 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.804281 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:51.805766 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.805740 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:15:51.805766 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.805754 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-kube-api-access-4h6nw" (OuterVolumeSpecName: "kube-api-access-4h6nw") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "kube-api-access-4h6nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:15:51.806045 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.806027 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:15:51.806098 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.806071 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0fcb6f1-3f5d-4667-93ed-141f67ea4853" (UID: "c0fcb6f1-3f5d-4667-93ed-141f67ea4853"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:15:51.904776 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.904735 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.904776 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.904770 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.904776 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.904784 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6nw\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-kube-api-access-4h6nw\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.905025 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.904797 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.905025 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.904808 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.905025 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.904819 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:51.905025 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:51.904830 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fcb6f1-3f5d-4667-93ed-141f67ea4853-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:15:52.277786 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.277757 2566 generic.go:356] "Generic (PLEG): container finished" podID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerID="252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5" exitCode=0 Jul 17 22:15:52.277964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.277804 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" Jul 17 22:15:52.277964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.277818 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" event={"ID":"c0fcb6f1-3f5d-4667-93ed-141f67ea4853","Type":"ContainerDied","Data":"252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5"} Jul 17 22:15:52.277964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.277885 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c" event={"ID":"c0fcb6f1-3f5d-4667-93ed-141f67ea4853","Type":"ContainerDied","Data":"587760d8849cac016fcce52bc58454fab4094cf38d9297006632aa52116ffd5d"} Jul 17 22:15:52.277964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.277906 2566 scope.go:117] "RemoveContainer" containerID="252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5" Jul 17 22:15:52.288391 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.288370 2566 scope.go:117] "RemoveContainer" containerID="5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b" Jul 17 22:15:52.296617 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.296595 2566 scope.go:117] "RemoveContainer" containerID="252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5" Jul 17 22:15:52.296870 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:15:52.296851 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5\": container with ID starting with 252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5 not found: ID does not exist" containerID="252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5" Jul 17 22:15:52.296944 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.296883 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5"} err="failed to get container status \"252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5\": rpc error: code = NotFound desc = could not find container \"252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5\": container with ID starting with 252ce645d6c1eb490011eae6ab06d3df586e827de72ef436eaca741920ff6ea5 not found: ID does not exist" Jul 17 22:15:52.296944 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.296909 2566 scope.go:117] "RemoveContainer" containerID="5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b" Jul 17 22:15:52.297167 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:15:52.297150 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b\": container with ID starting with 5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b not found: ID does not exist" containerID="5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b" Jul 17 22:15:52.297208 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.297174 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b"} err="failed to get container status \"5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b\": rpc error: code = NotFound desc = could not find container \"5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b\": container with ID starting with 5430cd858bda22fad18c00d50e2bdfde5cf51446532940a9741df13f2d7b359b not found: ID does not exist" Jul 17 22:15:52.302984 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.302961 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c"] Jul 17 22:15:52.307324 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:52.307302 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b09be9-openshift-default-549467bb45-zfx9c"] Jul 17 22:15:53.298571 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:53.298537 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" path="/var/lib/kubelet/pods/c0fcb6f1-3f5d-4667-93ed-141f67ea4853/volumes" Jul 17 22:15:55.493182 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.493146 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f"] Jul 17 22:15:55.493996 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.493971 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" Jul 17 22:15:55.493996 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.493995 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" Jul 17 22:15:55.494149 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.494064 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" Jul 17 22:15:55.494149 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.494072 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" Jul 17 22:15:55.500649 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.500628 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.503223 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.503204 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6fbd36-openshift-default-dockercfg-9glp5\"" Jul 17 22:15:55.511518 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.511496 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f"] Jul 17 22:15:55.641751 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641715 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-certs\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.641751 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-podinfo\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.641981 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641778 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-credential-socket\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.641981 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-envoy\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.641981 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641906 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0383998-f4b2-453c-ba9e-2aadbac331af-istiod-ca-cert\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.641981 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641933 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-socket\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.641981 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641957 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-token\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.642171 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.641995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdz86\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-kube-api-access-gdz86\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.642171 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.642053 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-data\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.685795 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.685757 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:15:55.743372 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743284 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-podinfo\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743372 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743326 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-credential-socket\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743586 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-envoy\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743586 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0383998-f4b2-453c-ba9e-2aadbac331af-istiod-ca-cert\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743586 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-socket\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743742 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-token\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743742 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743683 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdz86\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-kube-api-access-gdz86\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743742 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743710 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-data\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743906 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-credential-socket\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.743906 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.743757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-certs\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.744038 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.744016 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-certs\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.744401 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.744277 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0383998-f4b2-453c-ba9e-2aadbac331af-istiod-ca-cert\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.744527 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.744309 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-data\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.744634 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.744536 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-socket\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.745879 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.745858 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-envoy\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.746144 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.746126 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-podinfo\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.752873 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.752847 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdz86\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-kube-api-access-gdz86\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.752978 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.752952 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-token\") pod \"e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.812319 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.812284 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:55.981554 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:15:55.981519 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0383998_f4b2_453c_ba9e_2aadbac331af.slice/crio-44e3d1d8ea68b56aecb15959c76b5ce3b58849d02e457c50290ce812e9b14cb9 WatchSource:0}: Error finding container 44e3d1d8ea68b56aecb15959c76b5ce3b58849d02e457c50290ce812e9b14cb9: Status 404 returned error can't find the container with id 44e3d1d8ea68b56aecb15959c76b5ce3b58849d02e457c50290ce812e9b14cb9 Jul 17 22:15:55.981554 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.981539 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f"] Jul 17 22:15:55.983760 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.983721 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:15:55.983875 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.983797 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:15:55.983875 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:55.983833 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:15:56.303253 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:56.302439 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" event={"ID":"e0383998-f4b2-453c-ba9e-2aadbac331af","Type":"ContainerStarted","Data":"45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7"} Jul 17 22:15:56.303253 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:56.302503 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" event={"ID":"e0383998-f4b2-453c-ba9e-2aadbac331af","Type":"ContainerStarted","Data":"44e3d1d8ea68b56aecb15959c76b5ce3b58849d02e457c50290ce812e9b14cb9"} Jul 17 22:15:56.325325 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:56.325263 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" podStartSLOduration=1.325246693 podStartE2EDuration="1.325246693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:15:56.323000545 +0000 UTC m=+2893.592998261" watchObservedRunningTime="2026-07-17 22:15:56.325246693 +0000 UTC m=+2893.595244462" Jul 17 22:15:56.812674 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:56.812626 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:57.818022 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:57.817992 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:58.312299 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:58.312268 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:15:58.313444 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:15:58.313416 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:16:04.264151 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.264111 2566 patch_prober.go:28] interesting pod/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 22:16:04.264818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.264786 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 22:16:04.344118 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.344084 2566 generic.go:356] "Generic (PLEG): container finished" podID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerID="35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d" exitCode=0 Jul 17 22:16:04.344306 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.344165 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" event={"ID":"d7bc2a07-f612-4312-89ab-0bc77abbbd36","Type":"ContainerDied","Data":"35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d"} Jul 17 22:16:04.344635 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.344620 2566 scope.go:117] "RemoveContainer" containerID="35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d" Jul 17 22:16:04.345652 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.345620 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:16:04.345738 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.345690 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:16:04.345738 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:04.345722 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:16:05.353299 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:05.353219 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" event={"ID":"d7bc2a07-f612-4312-89ab-0bc77abbbd36","Type":"ContainerStarted","Data":"628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b"} Jul 17 22:16:05.679911 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:05.679867 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:16:05.680139 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:05.680007 2566 patch_prober.go:28] interesting pod/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 22:16:05.680139 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:05.680055 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 22:16:06.679885 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:06.679842 2566 patch_prober.go:28] interesting pod/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 22:16:06.680875 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:06.680479 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 22:16:07.680285 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:07.680222 2566 patch_prober.go:28] interesting pod/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 22:16:07.681391 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:07.680884 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 22:16:08.679613 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:08.679573 2566 patch_prober.go:28] interesting pod/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 17 22:16:08.679783 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:08.679645 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 17 22:16:09.685007 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:09.684975 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:16:09.685511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:09.685290 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:16:09.686141 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:09.686121 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:16:11.393404 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:11.393312 2566 generic.go:356] "Generic (PLEG): container finished" podID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerID="45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7" exitCode=0 Jul 17 22:16:11.393404 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:11.393397 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" event={"ID":"e0383998-f4b2-453c-ba9e-2aadbac331af","Type":"ContainerDied","Data":"45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7"} Jul 17 22:16:11.393926 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:11.393908 2566 scope.go:117] "RemoveContainer" containerID="45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7" Jul 17 22:16:11.394922 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:11.394894 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:16:11.394982 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:11.394968 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:16:11.395025 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:11.395002 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:16:12.400467 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:12.400435 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" event={"ID":"e0383998-f4b2-453c-ba9e-2aadbac331af","Type":"ContainerStarted","Data":"05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a"} Jul 17 22:16:12.513813 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:12.513765 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:16:12.813423 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:12.813390 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:16:12.814475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:12.814445 2566 patch_prober.go:28] interesting pod/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 22:16:12.814585 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:12.814501 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 22:16:13.312375 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:13.312346 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:16:13.813722 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:13.813688 2566 patch_prober.go:28] interesting pod/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 22:16:13.814112 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:13.813745 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 22:16:14.813671 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:14.813630 2566 patch_prober.go:28] interesting pod/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 22:16:14.813925 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:14.813691 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 22:16:15.812813 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:15.812771 2566 patch_prober.go:28] interesting pod/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 22:16:15.812996 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:15.812830 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 22:16:16.816581 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:16.816548 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:16:16.817602 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:16.817579 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:16:25.408779 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:25.408739 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:16:36.554288 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:36.554248 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z"] Jul 17 22:16:36.554756 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:36.554478 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" containerID="cri-o://628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b" gracePeriod=30 Jul 17 22:16:39.687110 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:39.687069 2566 patch_prober.go:28] interesting pod/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 22:16:39.687500 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:39.687129 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 22:16:41.803285 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.803258 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:16:41.898296 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898193 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-credential-socket\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898459 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898304 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-certs\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898459 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898350 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-envoy\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898459 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898381 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istiod-ca-cert\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898459 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898427 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-token\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898677 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898458 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-socket\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898677 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898494 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-podinfo\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898677 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898491 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:16:41.898677 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898516 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-data\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898677 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898582 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7v5k\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-kube-api-access-t7v5k\") pod \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\" (UID: \"d7bc2a07-f612-4312-89ab-0bc77abbbd36\") " Jul 17 22:16:41.898954 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898734 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:16:41.898954 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898817 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:16:41.898954 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898863 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:16:41.899082 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898954 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:41.899082 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898974 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:41.899082 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.898990 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:41.899082 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.899006 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:41.899664 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.899643 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:16:41.900838 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.900805 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:16:41.900969 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.900944 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-kube-api-access-t7v5k" (OuterVolumeSpecName: "kube-api-access-t7v5k") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "kube-api-access-t7v5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:16:41.901283 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.901257 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:16:41.901283 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.901223 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7bc2a07-f612-4312-89ab-0bc77abbbd36" (UID: "d7bc2a07-f612-4312-89ab-0bc77abbbd36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:16:42.000032 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:41.999997 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:42.000032 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.000029 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:42.000032 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.000038 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:42.000313 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.000046 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7bc2a07-f612-4312-89ab-0bc77abbbd36-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:42.000313 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.000055 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7v5k\" (UniqueName: \"kubernetes.io/projected/d7bc2a07-f612-4312-89ab-0bc77abbbd36-kube-api-access-t7v5k\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:16:42.557303 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.557269 2566 generic.go:356] "Generic (PLEG): container finished" podID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerID="628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b" exitCode=0 Jul 17 22:16:42.557475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.557329 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" Jul 17 22:16:42.557475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.557348 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" event={"ID":"d7bc2a07-f612-4312-89ab-0bc77abbbd36","Type":"ContainerDied","Data":"628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b"} Jul 17 22:16:42.557475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.557417 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z" event={"ID":"d7bc2a07-f612-4312-89ab-0bc77abbbd36","Type":"ContainerDied","Data":"728b3faecc8dcb4ae0cfed54245a758d1cd07aae5f27a28addd4adc78fc347c7"} Jul 17 22:16:42.557475 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.557440 2566 scope.go:117] "RemoveContainer" containerID="628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b" Jul 17 22:16:42.569303 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.569279 2566 scope.go:117] "RemoveContainer" containerID="35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d" Jul 17 22:16:42.578110 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.578077 2566 scope.go:117] "RemoveContainer" containerID="628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b" Jul 17 22:16:42.578436 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:16:42.578415 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b\": container with ID starting with 628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b not found: ID does not exist" containerID="628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b" Jul 17 22:16:42.578510 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.578446 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b"} err="failed to get container status \"628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b\": rpc error: code = NotFound desc = could not find container \"628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b\": container with ID starting with 628b6001f2565f564632e17bb985d3ab0b36e0a35cc66dcdced96390b1f92e7b not found: ID does not exist" Jul 17 22:16:42.578510 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.578466 2566 scope.go:117] "RemoveContainer" containerID="35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d" Jul 17 22:16:42.578736 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:16:42.578718 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d\": container with ID starting with 35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d not found: ID does not exist" containerID="35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d" Jul 17 22:16:42.578791 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.578746 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d"} err="failed to get container status \"35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d\": rpc error: code = NotFound desc = could not find container \"35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d\": container with ID starting with 35022d819160d38809de992e9b0e7d14b41dcfc15b91b30690b954e6e013964d not found: ID does not exist" Jul 17 22:16:42.579116 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.579096 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z"] Jul 17 22:16:42.582628 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:42.582606 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-95299c-openshift-default-6999645f65-2dp6z"] Jul 17 22:16:43.298626 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:43.298590 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" path="/var/lib/kubelet/pods/d7bc2a07-f612-4312-89ab-0bc77abbbd36/volumes" Jul 17 22:16:50.599584 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:16:50.599548 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:17:02.672507 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:02.672474 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f"] Jul 17 22:17:02.673000 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:02.672743 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" containerID="cri-o://05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a" gracePeriod=30 Jul 17 22:17:03.557736 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.557697 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms"] Jul 17 22:17:03.558072 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.558059 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" Jul 17 22:17:03.558124 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.558073 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fcb6f1-3f5d-4667-93ed-141f67ea4853" containerName="istio-proxy" Jul 17 22:17:03.558124 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.558093 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" Jul 17 22:17:03.558124 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.558099 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" Jul 17 22:17:03.558217 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.558154 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" Jul 17 22:17:03.561038 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.561019 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.563714 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.563689 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-8e6a3a-openshift-default-dockercfg-8msp4\"" Jul 17 22:17:03.575727 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.575699 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms"] Jul 17 22:17:03.594575 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594543 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-socket\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594709 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-credential-socket\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594709 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594596 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmhb\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-kube-api-access-tsmhb\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594709 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594614 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-podinfo\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594709 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594701 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-envoy\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594904 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-data\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594904 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-certs\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594904 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594748 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-token\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.594904 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.594773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9925a74d-c3a1-4935-aabc-485b8d43195a-istiod-ca-cert\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.695850 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.695813 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-envoy\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.695850 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.695855 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-data\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.695874 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-certs\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.695898 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-token\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.695930 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9925a74d-c3a1-4935-aabc-485b8d43195a-istiod-ca-cert\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.695970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-socket\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.695992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-credential-socket\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.696018 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmhb\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-kube-api-access-tsmhb\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.696048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-podinfo\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696836 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.696442 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-certs\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696836 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.696476 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-socket\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696836 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.696573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-credential-socket\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.696836 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.696754 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-data\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.697168 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.697005 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9925a74d-c3a1-4935-aabc-485b8d43195a-istiod-ca-cert\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.698349 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.698329 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-envoy\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.698731 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.698707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-podinfo\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.704865 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.704830 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-token\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.704981 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.704951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmhb\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-kube-api-access-tsmhb\") pod \"e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:03.872522 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:03.872412 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:04.017295 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.017264 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms"] Jul 17 22:17:04.018308 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:17:04.018272 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9925a74d_c3a1_4935_aabc_485b8d43195a.slice/crio-6b59548f67ebb8ff22903dc5ad975e16ed65b3e9127c56cec546674325b93efe WatchSource:0}: Error finding container 6b59548f67ebb8ff22903dc5ad975e16ed65b3e9127c56cec546674325b93efe: Status 404 returned error can't find the container with id 6b59548f67ebb8ff22903dc5ad975e16ed65b3e9127c56cec546674325b93efe Jul 17 22:17:04.021139 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.021104 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:04.021270 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.021173 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:04.021270 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.021200 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:04.675855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.675810 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" event={"ID":"9925a74d-c3a1-4935-aabc-485b8d43195a","Type":"ContainerStarted","Data":"74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11"} Jul 17 22:17:04.676041 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.675873 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" event={"ID":"9925a74d-c3a1-4935-aabc-485b8d43195a","Type":"ContainerStarted","Data":"6b59548f67ebb8ff22903dc5ad975e16ed65b3e9127c56cec546674325b93efe"} Jul 17 22:17:04.696769 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.696712 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" podStartSLOduration=1.6966629050000002 podStartE2EDuration="1.696662905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:04.696164172 +0000 UTC m=+2961.966161900" watchObservedRunningTime="2026-07-17 22:17:04.696662905 +0000 UTC m=+2961.966660637" Jul 17 22:17:04.872656 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:04.872625 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:05.878421 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:05.878385 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:06.685813 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:06.685778 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:06.686871 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:06.686851 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:07.924467 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:07.924443 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:17:08.034336 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034214 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0383998-f4b2-453c-ba9e-2aadbac331af-istiod-ca-cert\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034336 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034323 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-token\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034572 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034353 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-credential-socket\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034572 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034374 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdz86\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-kube-api-access-gdz86\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034572 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034411 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-certs\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034572 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034466 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-podinfo\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034572 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034487 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-data\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034572 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034545 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-envoy\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034572 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034571 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-socket\") pod \"e0383998-f4b2-453c-ba9e-2aadbac331af\" (UID: \"e0383998-f4b2-453c-ba9e-2aadbac331af\") " Jul 17 22:17:08.034931 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034647 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0383998-f4b2-453c-ba9e-2aadbac331af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:17:08.034931 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034673 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:17:08.034931 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034756 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:17:08.034931 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.034901 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:17:08.035136 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.035034 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.035136 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.035052 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.035136 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.035067 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.035136 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.035080 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0383998-f4b2-453c-ba9e-2aadbac331af-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.035653 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.035620 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:17:08.036905 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.036874 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:17:08.036905 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.036888 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-kube-api-access-gdz86" (OuterVolumeSpecName: "kube-api-access-gdz86") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "kube-api-access-gdz86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:17:08.036905 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.036889 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:17:08.037108 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.037013 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0383998-f4b2-453c-ba9e-2aadbac331af" (UID: "e0383998-f4b2-453c-ba9e-2aadbac331af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:17:08.136544 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.136508 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.136544 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.136537 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.136544 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.136547 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdz86\" (UniqueName: \"kubernetes.io/projected/e0383998-f4b2-453c-ba9e-2aadbac331af-kube-api-access-gdz86\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.136786 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.136557 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.136786 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.136566 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0383998-f4b2-453c-ba9e-2aadbac331af-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:17:08.698759 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.698720 2566 generic.go:356] "Generic (PLEG): container finished" podID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerID="05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a" exitCode=0 Jul 17 22:17:08.698964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.698775 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" Jul 17 22:17:08.698964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.698813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" event={"ID":"e0383998-f4b2-453c-ba9e-2aadbac331af","Type":"ContainerDied","Data":"05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a"} Jul 17 22:17:08.698964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.698867 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f" event={"ID":"e0383998-f4b2-453c-ba9e-2aadbac331af","Type":"ContainerDied","Data":"44e3d1d8ea68b56aecb15959c76b5ce3b58849d02e457c50290ce812e9b14cb9"} Jul 17 22:17:08.698964 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.698889 2566 scope.go:117] "RemoveContainer" containerID="05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a" Jul 17 22:17:08.711837 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.711678 2566 scope.go:117] "RemoveContainer" containerID="45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7" Jul 17 22:17:08.720313 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.720288 2566 scope.go:117] "RemoveContainer" containerID="05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a" Jul 17 22:17:08.720557 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:17:08.720541 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a\": container with ID starting with 05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a not found: ID does not exist" containerID="05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a" Jul 17 22:17:08.720615 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.720565 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a"} err="failed to get container status \"05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a\": rpc error: code = NotFound desc = could not find container \"05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a\": container with ID starting with 05feb3116ec41384f8e160da089c1948cb664ad69432c7f25b86f2af129f289a not found: ID does not exist" Jul 17 22:17:08.720615 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.720580 2566 scope.go:117] "RemoveContainer" containerID="45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7" Jul 17 22:17:08.720838 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:17:08.720819 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7\": container with ID starting with 45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7 not found: ID does not exist" containerID="45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7" Jul 17 22:17:08.720888 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.720842 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7"} err="failed to get container status \"45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7\": rpc error: code = NotFound desc = could not find container \"45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7\": container with ID starting with 45c1287385d3ae9da0029d924b36332f6b287f3f2cd36a55ad05f1d67dc416c7 not found: ID does not exist" Jul 17 22:17:08.725864 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.725844 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f"] Jul 17 22:17:08.729285 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:08.729264 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6fbd36-openshift-default-5f6fb97b4c-f4f9f"] Jul 17 22:17:09.298388 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:09.298354 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" path="/var/lib/kubelet/pods/e0383998-f4b2-453c-ba9e-2aadbac331af/volumes" Jul 17 22:17:11.609883 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.609855 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:17:11.762816 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.762773 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2"] Jul 17 22:17:11.763324 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.763288 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" Jul 17 22:17:11.763324 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.763314 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" Jul 17 22:17:11.763511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.763334 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" Jul 17 22:17:11.763511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.763344 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" Jul 17 22:17:11.763511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.763472 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7bc2a07-f612-4312-89ab-0bc77abbbd36" containerName="istio-proxy" Jul 17 22:17:11.763511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.763487 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" Jul 17 22:17:11.763511 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.763497 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" Jul 17 22:17:11.768107 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.768084 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.770495 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.770478 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5401e5-openshift-default-dockercfg-g9rck\"" Jul 17 22:17:11.778145 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.778125 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2"] Jul 17 22:17:11.872640 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872556 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-socket\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.872640 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872590 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-podinfo\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.872640 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872614 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-envoy\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.872866 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872690 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-data\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.872866 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872727 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcvlf\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-kube-api-access-mcvlf\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.872866 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-credential-socket\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.872866 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872802 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b42d8-7fe3-443a-9d23-1089b855fb33-istiod-ca-cert\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.872866 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872831 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-certs\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.873050 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.872889 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-token\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974215 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcvlf\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-kube-api-access-mcvlf\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974426 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974220 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-credential-socket\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974426 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b42d8-7fe3-443a-9d23-1089b855fb33-istiod-ca-cert\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974426 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-certs\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974426 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974368 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-token\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974426 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974426 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-socket\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974701 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974446 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-podinfo\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974701 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974474 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-envoy\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974701 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974526 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-data\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974701 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974632 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-credential-socket\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974919 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974700 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-certs\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974979 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974930 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-data\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.974979 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.974969 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-socket\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.975165 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.975145 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b42d8-7fe3-443a-9d23-1089b855fb33-istiod-ca-cert\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.976930 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.976909 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-envoy\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.977055 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.977037 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-podinfo\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.990198 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.990171 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-token\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:11.990398 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:11.990346 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcvlf\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-kube-api-access-mcvlf\") pod \"e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:12.079352 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.079317 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:12.222703 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.222677 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2"] Jul 17 22:17:12.224733 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:17:12.224704 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306b42d8_7fe3_443a_9d23_1089b855fb33.slice/crio-ea51c5971197cb36b0a13645c7c4f361990b8870862c0be9e2974960768e5d5f WatchSource:0}: Error finding container ea51c5971197cb36b0a13645c7c4f361990b8870862c0be9e2974960768e5d5f: Status 404 returned error can't find the container with id ea51c5971197cb36b0a13645c7c4f361990b8870862c0be9e2974960768e5d5f Jul 17 22:17:12.226866 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.226831 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:12.226960 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.226903 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:12.226960 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.226933 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:12.721254 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.721200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" event={"ID":"306b42d8-7fe3-443a-9d23-1089b855fb33","Type":"ContainerStarted","Data":"b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f"} Jul 17 22:17:12.721620 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.721271 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" event={"ID":"306b42d8-7fe3-443a-9d23-1089b855fb33","Type":"ContainerStarted","Data":"ea51c5971197cb36b0a13645c7c4f361990b8870862c0be9e2974960768e5d5f"} Jul 17 22:17:12.741921 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:12.741867 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" podStartSLOduration=1.7418477879999998 podStartE2EDuration="1.741847788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:12.740715286 +0000 UTC m=+2970.010713036" watchObservedRunningTime="2026-07-17 22:17:12.741847788 +0000 UTC m=+2970.011845546" Jul 17 22:17:13.079575 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:13.079479 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:13.084823 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:13.084794 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:13.726586 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:13.726543 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:13.727508 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:13.727484 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:20.771539 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:20.771501 2566 generic.go:356] "Generic (PLEG): container finished" podID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerID="74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11" exitCode=0 Jul 17 22:17:20.772094 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:20.771580 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" event={"ID":"9925a74d-c3a1-4935-aabc-485b8d43195a","Type":"ContainerDied","Data":"74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11"} Jul 17 22:17:20.772094 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:20.772075 2566 scope.go:117] "RemoveContainer" containerID="74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11" Jul 17 22:17:20.773415 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:20.773372 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:20.773550 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:20.773463 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:20.773550 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:20.773510 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:21.685873 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:21.685834 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:21.781619 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:21.781573 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" event={"ID":"9925a74d-c3a1-4935-aabc-485b8d43195a","Type":"ContainerStarted","Data":"8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871"} Jul 17 22:17:21.873755 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:21.873721 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:21.874714 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:21.874681 2566 patch_prober.go:28] interesting pod/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 17 22:17:21.874857 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:21.874740 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 17 22:17:22.873078 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:22.873019 2566 patch_prober.go:28] interesting pod/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 17 22:17:22.873594 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:22.873100 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 17 22:17:23.873181 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:23.873140 2566 patch_prober.go:28] interesting pod/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 17 22:17:23.873824 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:23.873788 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 17 22:17:24.873777 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:24.873695 2566 patch_prober.go:28] interesting pod/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 17 22:17:24.873777 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:24.873771 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 17 22:17:25.872893 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:25.872846 2566 patch_prober.go:28] interesting pod/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 17 22:17:25.873075 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:25.872925 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 17 22:17:26.876826 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:26.876791 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:26.877333 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:26.877183 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:26.877887 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:26.877863 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:17:27.818573 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:27.818538 2566 generic.go:356] "Generic (PLEG): container finished" podID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerID="b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f" exitCode=0 Jul 17 22:17:27.818823 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:27.818610 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" event={"ID":"306b42d8-7fe3-443a-9d23-1089b855fb33","Type":"ContainerDied","Data":"b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f"} Jul 17 22:17:27.819145 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:27.819128 2566 scope.go:117] "RemoveContainer" containerID="b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f" Jul 17 22:17:27.820076 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:27.820048 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:27.820171 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:27.820119 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:27.820171 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:27.820155 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 22:17:28.726868 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:28.726823 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:28.824980 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:28.824943 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" event={"ID":"306b42d8-7fe3-443a-9d23-1089b855fb33","Type":"ContainerStarted","Data":"bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e"} Jul 17 22:17:29.080248 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:29.080127 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:29.081432 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:29.081407 2566 patch_prober.go:28] interesting pod/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 22:17:29.081494 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:29.081458 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 22:17:30.080144 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:30.080103 2566 patch_prober.go:28] interesting pod/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 22:17:30.080573 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:30.080159 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 22:17:30.210602 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:30.210567 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:17:31.080342 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:31.080304 2566 patch_prober.go:28] interesting pod/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 22:17:31.080748 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:31.080366 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 22:17:32.080244 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:32.080195 2566 patch_prober.go:28] interesting pod/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 17 22:17:32.080424 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:32.080269 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 17 22:17:33.083731 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:33.083700 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:33.084071 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:33.084036 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:33.084604 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:33.084587 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:17:48.142559 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:17:48.142519 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:18:10.506900 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:18:10.506866 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:18:29.507407 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:18:29.507368 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:18:47.904281 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:18:47.904219 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:19:02.704946 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:02.704909 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:19:17.898017 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:17.897982 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:19:35.605685 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:35.605649 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:19:50.706626 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:50.706589 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:19:53.837729 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.837698 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw"] Jul 17 22:19:53.838125 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.838100 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" Jul 17 22:19:53.838125 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.838113 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0383998-f4b2-453c-ba9e-2aadbac331af" containerName="istio-proxy" Jul 17 22:19:53.840134 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.840116 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" Jul 17 22:19:53.842649 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.842630 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:19:53.842784 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.842633 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:19:53.848192 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.848033 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw"] Jul 17 22:19:53.941428 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:53.941393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a933f794-2b24-483f-b58a-b68f8f2408ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw\" (UID: \"a933f794-2b24-483f-b58a-b68f8f2408ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" Jul 17 22:19:54.042346 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:54.042295 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a933f794-2b24-483f-b58a-b68f8f2408ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw\" (UID: \"a933f794-2b24-483f-b58a-b68f8f2408ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" Jul 17 22:19:54.042946 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:54.042924 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a933f794-2b24-483f-b58a-b68f8f2408ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw\" (UID: \"a933f794-2b24-483f-b58a-b68f8f2408ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" Jul 17 22:19:54.153942 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:54.153842 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" Jul 17 22:19:54.301222 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:54.301198 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw"] Jul 17 22:19:54.303511 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:19:54.303485 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda933f794_2b24_483f_b58a_b68f8f2408ae.slice/crio-de01a76036f09f7fdeba859dc797243a0b6b535576898e073b837cb90de81aab WatchSource:0}: Error finding container de01a76036f09f7fdeba859dc797243a0b6b535576898e073b837cb90de81aab: Status 404 returned error can't find the container with id de01a76036f09f7fdeba859dc797243a0b6b535576898e073b837cb90de81aab Jul 17 22:19:54.551951 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:54.551908 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" event={"ID":"a933f794-2b24-483f-b58a-b68f8f2408ae","Type":"ContainerStarted","Data":"de01a76036f09f7fdeba859dc797243a0b6b535576898e073b837cb90de81aab"} Jul 17 22:19:55.559077 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:55.559035 2566 generic.go:356] "Generic (PLEG): container finished" podID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerID="ca79546b4602dc42db6545d6bd0036f3ffde743e1e33ef478627e2012f78ce66" exitCode=6 Jul 17 22:19:55.559556 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:55.559082 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" event={"ID":"a933f794-2b24-483f-b58a-b68f8f2408ae","Type":"ContainerDied","Data":"ca79546b4602dc42db6545d6bd0036f3ffde743e1e33ef478627e2012f78ce66"} Jul 17 22:19:55.559556 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:55.559419 2566 scope.go:117] "RemoveContainer" containerID="ca79546b4602dc42db6545d6bd0036f3ffde743e1e33ef478627e2012f78ce66" Jul 17 22:19:56.566267 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:56.566160 2566 generic.go:356] "Generic (PLEG): container finished" podID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerID="fbd555bfe89af4b17530d3f63c8c4119707b3f2cc9ec1f3bc3ae205ba23fd3ee" exitCode=6 Jul 17 22:19:56.566267 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:56.566218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" event={"ID":"a933f794-2b24-483f-b58a-b68f8f2408ae","Type":"ContainerDied","Data":"fbd555bfe89af4b17530d3f63c8c4119707b3f2cc9ec1f3bc3ae205ba23fd3ee"} Jul 17 22:19:56.566740 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:56.566313 2566 scope.go:117] "RemoveContainer" containerID="ca79546b4602dc42db6545d6bd0036f3ffde743e1e33ef478627e2012f78ce66" Jul 17 22:19:56.566740 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:56.566456 2566 scope.go:117] "RemoveContainer" containerID="fbd555bfe89af4b17530d3f63c8c4119707b3f2cc9ec1f3bc3ae205ba23fd3ee" Jul 17 22:19:56.566740 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:19:56.566714 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw_odh-ai-gateway-infra(a933f794-2b24-483f-b58a-b68f8f2408ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" Jul 17 22:19:57.574060 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:19:57.574029 2566 scope.go:117] "RemoveContainer" containerID="fbd555bfe89af4b17530d3f63c8c4119707b3f2cc9ec1f3bc3ae205ba23fd3ee" Jul 17 22:19:57.574478 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:19:57.574214 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw_odh-ai-gateway-infra(a933f794-2b24-483f-b58a-b68f8f2408ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" Jul 17 22:20:06.606742 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:06.606661 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:20:09.292498 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:09.292465 2566 scope.go:117] "RemoveContainer" containerID="fbd555bfe89af4b17530d3f63c8c4119707b3f2cc9ec1f3bc3ae205ba23fd3ee" Jul 17 22:20:09.633060 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:09.633026 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" event={"ID":"a933f794-2b24-483f-b58a-b68f8f2408ae","Type":"ContainerStarted","Data":"c0ca9202561bfc0f262d2f296eb4c1074021cee073dafa28bda55f8213501267"} Jul 17 22:20:09.648973 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:09.648924 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" podStartSLOduration=16.648904664 podStartE2EDuration="16.648904664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:20:09.648323303 +0000 UTC m=+3146.918321029" watchObservedRunningTime="2026-07-17 22:20:09.648904664 +0000 UTC m=+3146.918902387" Jul 17 22:20:10.640494 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:10.640461 2566 generic.go:356] "Generic (PLEG): container finished" podID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerID="c0ca9202561bfc0f262d2f296eb4c1074021cee073dafa28bda55f8213501267" exitCode=6 Jul 17 22:20:10.640921 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:10.640519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" event={"ID":"a933f794-2b24-483f-b58a-b68f8f2408ae","Type":"ContainerDied","Data":"c0ca9202561bfc0f262d2f296eb4c1074021cee073dafa28bda55f8213501267"} Jul 17 22:20:10.640921 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:10.640603 2566 scope.go:117] "RemoveContainer" containerID="fbd555bfe89af4b17530d3f63c8c4119707b3f2cc9ec1f3bc3ae205ba23fd3ee" Jul 17 22:20:10.640921 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:10.640871 2566 scope.go:117] "RemoveContainer" containerID="c0ca9202561bfc0f262d2f296eb4c1074021cee073dafa28bda55f8213501267" Jul 17 22:20:10.641128 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:20:10.641107 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw_odh-ai-gateway-infra(a933f794-2b24-483f-b58a-b68f8f2408ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" Jul 17 22:20:10.660669 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:10.660644 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw"] Jul 17 22:20:11.789326 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:11.789299 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" Jul 17 22:20:11.913957 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:11.913930 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a933f794-2b24-483f-b58a-b68f8f2408ae-maas-api-service-ca\") pod \"a933f794-2b24-483f-b58a-b68f8f2408ae\" (UID: \"a933f794-2b24-483f-b58a-b68f8f2408ae\") " Jul 17 22:20:11.914311 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:11.914288 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a933f794-2b24-483f-b58a-b68f8f2408ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a933f794-2b24-483f-b58a-b68f8f2408ae" (UID: "a933f794-2b24-483f-b58a-b68f8f2408ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:20:12.015349 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:12.015315 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a933f794-2b24-483f-b58a-b68f8f2408ae-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:20:12.655252 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:12.655197 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" Jul 17 22:20:12.655252 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:12.655209 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw" event={"ID":"a933f794-2b24-483f-b58a-b68f8f2408ae","Type":"ContainerDied","Data":"de01a76036f09f7fdeba859dc797243a0b6b535576898e073b837cb90de81aab"} Jul 17 22:20:12.655529 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:12.655303 2566 scope.go:117] "RemoveContainer" containerID="c0ca9202561bfc0f262d2f296eb4c1074021cee073dafa28bda55f8213501267" Jul 17 22:20:12.678872 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:12.678842 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw"] Jul 17 22:20:12.683299 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:12.683277 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-nt5fw"] Jul 17 22:20:13.298534 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:13.298492 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" path="/var/lib/kubelet/pods/a933f794-2b24-483f-b58a-b68f8f2408ae/volumes" Jul 17 22:20:23.204916 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:23.204877 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:20:38.115855 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:38.115816 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:20:58.113770 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:20:58.113733 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:21:17.205357 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:21:17.205317 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:21:38.016344 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:21:38.016304 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:22:00.212601 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:22:00.212556 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:22:27.900564 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:22:27.900533 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:22:43.300506 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:22:43.300471 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:22:57.801379 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:22:57.801327 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:23:23.999836 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:23.999761 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:23:35.980623 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:35.980589 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms"] Jul 17 22:23:35.981549 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:35.980810 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" containerID="cri-o://8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871" gracePeriod=30 Jul 17 22:23:41.245384 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.245317 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:23:41.303619 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303589 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-credential-socket\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.303619 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303621 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-data\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.303846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303679 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-envoy\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.303846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303704 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-token\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.303846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303726 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsmhb\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-kube-api-access-tsmhb\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.303846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303748 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-certs\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.303846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303804 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-podinfo\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.303846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303848 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-socket\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.304146 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303867 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:23:41.304146 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.303977 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9925a74d-c3a1-4935-aabc-485b8d43195a-istiod-ca-cert\") pod \"9925a74d-c3a1-4935-aabc-485b8d43195a\" (UID: \"9925a74d-c3a1-4935-aabc-485b8d43195a\") " Jul 17 22:23:41.304284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304131 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:23:41.304284 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304150 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:23:41.304565 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304543 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:23:41.304699 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304680 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.304741 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304715 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.304741 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304730 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.304805 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304746 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.304805 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.304680 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9925a74d-c3a1-4935-aabc-485b8d43195a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:23:41.306105 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.306077 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:41.306271 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.306214 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-kube-api-access-tsmhb" (OuterVolumeSpecName: "kube-api-access-tsmhb") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "kube-api-access-tsmhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:41.306574 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.306555 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:23:41.306704 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.306668 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9925a74d-c3a1-4935-aabc-485b8d43195a" (UID: "9925a74d-c3a1-4935-aabc-485b8d43195a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:23:41.405856 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.405817 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.405856 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.405849 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.405856 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.405860 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsmhb\" (UniqueName: \"kubernetes.io/projected/9925a74d-c3a1-4935-aabc-485b8d43195a-kube-api-access-tsmhb\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.406096 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.405870 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9925a74d-c3a1-4935-aabc-485b8d43195a-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.406096 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.405880 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9925a74d-c3a1-4935-aabc-485b8d43195a-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:23:41.691300 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.691259 2566 generic.go:356] "Generic (PLEG): container finished" podID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerID="8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871" exitCode=0 Jul 17 22:23:41.691467 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.691311 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" Jul 17 22:23:41.691467 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.691320 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" event={"ID":"9925a74d-c3a1-4935-aabc-485b8d43195a","Type":"ContainerDied","Data":"8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871"} Jul 17 22:23:41.691467 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.691379 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms" event={"ID":"9925a74d-c3a1-4935-aabc-485b8d43195a","Type":"ContainerDied","Data":"6b59548f67ebb8ff22903dc5ad975e16ed65b3e9127c56cec546674325b93efe"} Jul 17 22:23:41.691467 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.691401 2566 scope.go:117] "RemoveContainer" containerID="8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871" Jul 17 22:23:41.701024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.701008 2566 scope.go:117] "RemoveContainer" containerID="74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11" Jul 17 22:23:41.709381 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.709360 2566 scope.go:117] "RemoveContainer" containerID="8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871" Jul 17 22:23:41.709642 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:23:41.709621 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871\": container with ID starting with 8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871 not found: ID does not exist" containerID="8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871" Jul 17 22:23:41.709682 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.709658 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871"} err="failed to get container status \"8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871\": rpc error: code = NotFound desc = could not find container \"8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871\": container with ID starting with 8545ca0502de1a1285f2c925d87a5dbf104e2bc17f31cf223560703b03014871 not found: ID does not exist" Jul 17 22:23:41.709682 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.709679 2566 scope.go:117] "RemoveContainer" containerID="74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11" Jul 17 22:23:41.709901 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:23:41.709884 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11\": container with ID starting with 74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11 not found: ID does not exist" containerID="74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11" Jul 17 22:23:41.709940 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.709908 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11"} err="failed to get container status \"74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11\": rpc error: code = NotFound desc = could not find container \"74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11\": container with ID starting with 74488d7907e769f189555660ea0a32821f7bbc8195aaf83fecb3229c2ccbdc11 not found: ID does not exist" Jul 17 22:23:41.713949 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.713927 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms"] Jul 17 22:23:41.717656 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:41.717634 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-8e6a3a-openshift-default-586bcc6c87-9h8ms"] Jul 17 22:23:43.298213 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:43.298174 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" path="/var/lib/kubelet/pods/9925a74d-c3a1-4935-aabc-485b8d43195a/volumes" Jul 17 22:23:49.702833 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:23:49.702795 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jxrtm"] Jul 17 22:24:02.534309 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:02.534275 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2"] Jul 17 22:24:02.534752 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:02.534503 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" containerID="cri-o://bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e" gracePeriod=30 Jul 17 22:24:03.086842 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:03.086799 2566 patch_prober.go:28] interesting pod/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 22:24:03.087029 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:03.086885 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 22:24:07.792447 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.792423 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:24:07.821172 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.821080 2566 generic.go:356] "Generic (PLEG): container finished" podID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerID="bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e" exitCode=0 Jul 17 22:24:07.821172 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.821141 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" Jul 17 22:24:07.821433 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.821176 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" event={"ID":"306b42d8-7fe3-443a-9d23-1089b855fb33","Type":"ContainerDied","Data":"bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e"} Jul 17 22:24:07.821433 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.821265 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2" event={"ID":"306b42d8-7fe3-443a-9d23-1089b855fb33","Type":"ContainerDied","Data":"ea51c5971197cb36b0a13645c7c4f361990b8870862c0be9e2974960768e5d5f"} Jul 17 22:24:07.821433 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.821300 2566 scope.go:117] "RemoveContainer" containerID="bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e" Jul 17 22:24:07.830458 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.830442 2566 scope.go:117] "RemoveContainer" containerID="b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f" Jul 17 22:24:07.838399 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.838377 2566 scope.go:117] "RemoveContainer" containerID="bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e" Jul 17 22:24:07.838655 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:07.838638 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e\": container with ID starting with bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e not found: ID does not exist" containerID="bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e" Jul 17 22:24:07.838723 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.838665 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e"} err="failed to get container status \"bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e\": rpc error: code = NotFound desc = could not find container \"bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e\": container with ID starting with bd836ac6777411dfd8152c868b5352bdf7c2d1b799007c2e6da5debb8d32c09e not found: ID does not exist" Jul 17 22:24:07.838723 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.838681 2566 scope.go:117] "RemoveContainer" containerID="b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f" Jul 17 22:24:07.838973 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:07.838954 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f\": container with ID starting with b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f not found: ID does not exist" containerID="b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f" Jul 17 22:24:07.839030 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.838980 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f"} err="failed to get container status \"b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f\": rpc error: code = NotFound desc = could not find container \"b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f\": container with ID starting with b7deacbe4d5186e19bd8eee4c87617304cc9992ddc0d7066e2fae1f632ef255f not found: ID does not exist" Jul 17 22:24:07.951850 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.951822 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-certs\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.951861 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-data\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.951922 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-token\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.951948 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcvlf\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-kube-api-access-mcvlf\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.951969 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-podinfo\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.951985 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b42d8-7fe3-443a-9d23-1089b855fb33-istiod-ca-cert\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952024 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.952005 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-credential-socket\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952356 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.952045 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-envoy\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952356 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.952189 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:24:07.952356 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.952276 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-socket\") pod \"306b42d8-7fe3-443a-9d23-1089b855fb33\" (UID: \"306b42d8-7fe3-443a-9d23-1089b855fb33\") " Jul 17 22:24:07.952621 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.952592 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-certs\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:07.952786 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.952768 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-data" (OuterVolumeSpecName: "istio-data") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:24:07.953018 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.952998 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:24:07.953137 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.953058 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306b42d8-7fe3-443a-9d23-1089b855fb33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:24:07.953494 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.953470 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:24:07.954558 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.954531 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-kube-api-access-mcvlf" (OuterVolumeSpecName: "kube-api-access-mcvlf") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "kube-api-access-mcvlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:24:07.954782 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.954756 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 22:24:07.954905 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.954869 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-token" (OuterVolumeSpecName: "istio-token") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:24:07.955205 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:07.955188 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "306b42d8-7fe3-443a-9d23-1089b855fb33" (UID: "306b42d8-7fe3-443a-9d23-1089b855fb33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:24:08.053092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053051 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-podinfo\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.053092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053083 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/306b42d8-7fe3-443a-9d23-1089b855fb33-istiod-ca-cert\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.053092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053095 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-credential-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.053092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053105 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-envoy\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.053396 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053114 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-workload-socket\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.053396 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053123 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-data\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.053396 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053131 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-istio-token\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.053396 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.053139 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcvlf\" (UniqueName: \"kubernetes.io/projected/306b42d8-7fe3-443a-9d23-1089b855fb33-kube-api-access-mcvlf\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:08.143434 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.143396 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2"] Jul 17 22:24:08.146822 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.146793 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5401e5-openshift-default-57959dd7ff-85nf2"] Jul 17 22:24:08.414391 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414352 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6"] Jul 17 22:24:08.414779 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414761 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:08.414878 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414782 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:08.414878 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414796 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" Jul 17 22:24:08.414878 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414804 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" Jul 17 22:24:08.414878 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414816 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" Jul 17 22:24:08.414878 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414824 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" Jul 17 22:24:08.414878 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414853 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" Jul 17 22:24:08.414878 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414861 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414881 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414890 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.414992 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.415005 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.415014 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.415025 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.415034 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" Jul 17 22:24:08.415246 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.415043 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" containerName="istio-proxy" Jul 17 22:24:08.419411 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.419389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" Jul 17 22:24:08.421932 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.421906 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 22:24:08.422059 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.421968 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dnrgc\"" Jul 17 22:24:08.425341 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.425315 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6"] Jul 17 22:24:08.557094 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.557053 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8612b4-7c51-4322-9ae8-f0f53d2dd480-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6\" (UID: \"7e8612b4-7c51-4322-9ae8-f0f53d2dd480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" Jul 17 22:24:08.657736 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.657692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8612b4-7c51-4322-9ae8-f0f53d2dd480-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6\" (UID: \"7e8612b4-7c51-4322-9ae8-f0f53d2dd480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" Jul 17 22:24:08.658371 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.658351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8612b4-7c51-4322-9ae8-f0f53d2dd480-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6\" (UID: \"7e8612b4-7c51-4322-9ae8-f0f53d2dd480\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" Jul 17 22:24:08.733014 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.732908 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" Jul 17 22:24:08.867818 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:08.867778 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6"] Jul 17 22:24:08.870400 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:24:08.870369 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8612b4_7c51_4322_9ae8_f0f53d2dd480.slice/crio-844ca605f5172c3e2944d2695dced00911fd2ea0d273fadbb884ea7c3b0cf31e WatchSource:0}: Error finding container 844ca605f5172c3e2944d2695dced00911fd2ea0d273fadbb884ea7c3b0cf31e: Status 404 returned error can't find the container with id 844ca605f5172c3e2944d2695dced00911fd2ea0d273fadbb884ea7c3b0cf31e Jul 17 22:24:09.107375 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.107342 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m5p9g_e9da3247-ff20-4211-ad5f-640426eb1a6d/manager/0.log" Jul 17 22:24:09.298286 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.298250 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="306b42d8-7fe3-443a-9d23-1089b855fb33" path="/var/lib/kubelet/pods/306b42d8-7fe3-443a-9d23-1089b855fb33/volumes" Jul 17 22:24:09.833966 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.833882 2566 generic.go:356] "Generic (PLEG): container finished" podID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerID="dfdbd4fa0559ceda17f8e1e3a4de1f4df9acae759c3933199e028dbb75392558" exitCode=6 Jul 17 22:24:09.834129 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.833975 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" event={"ID":"7e8612b4-7c51-4322-9ae8-f0f53d2dd480","Type":"ContainerDied","Data":"dfdbd4fa0559ceda17f8e1e3a4de1f4df9acae759c3933199e028dbb75392558"} Jul 17 22:24:09.834129 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.834033 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" event={"ID":"7e8612b4-7c51-4322-9ae8-f0f53d2dd480","Type":"ContainerStarted","Data":"844ca605f5172c3e2944d2695dced00911fd2ea0d273fadbb884ea7c3b0cf31e"} Jul 17 22:24:09.834259 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.834136 2566 scope.go:117] "RemoveContainer" containerID="dfdbd4fa0559ceda17f8e1e3a4de1f4df9acae759c3933199e028dbb75392558" Jul 17 22:24:09.851977 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.851950 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-v2w56_48cf6d79-d7e0-451b-a3c6-53f54049e3b3/copy-manifests/0.log" Jul 17 22:24:09.859112 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:09.859093 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-v2w56_48cf6d79-d7e0-451b-a3c6-53f54049e3b3/manager/0.log" Jul 17 22:24:10.332598 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.332572 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738745-5xhxb_0bab2e9d-a935-4bca-a95f-f47662745737/cleanup/0.log" Jul 17 22:24:10.631425 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.631335 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq"] Jul 17 22:24:10.631727 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.631714 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:10.631787 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.631729 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:10.631787 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.631746 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" Jul 17 22:24:10.631787 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.631752 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9925a74d-c3a1-4935-aabc-485b8d43195a" containerName="istio-proxy" Jul 17 22:24:10.631890 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.631815 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a933f794-2b24-483f-b58a-b68f8f2408ae" containerName="revoke-keys" Jul 17 22:24:10.634726 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.634704 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" Jul 17 22:24:10.641157 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.641073 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq"] Jul 17 22:24:10.778724 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.778690 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a1cabb-4f72-4246-97ae-d5b23b10e394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq\" (UID: \"a7a1cabb-4f72-4246-97ae-d5b23b10e394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" Jul 17 22:24:10.840640 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.840612 2566 generic.go:356] "Generic (PLEG): container finished" podID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerID="0b1a9ee9a6a03c6dd27edc75b474a6e8a84a06855c35e30b149721c730da7484" exitCode=6 Jul 17 22:24:10.840788 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.840702 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" event={"ID":"7e8612b4-7c51-4322-9ae8-f0f53d2dd480","Type":"ContainerDied","Data":"0b1a9ee9a6a03c6dd27edc75b474a6e8a84a06855c35e30b149721c730da7484"} Jul 17 22:24:10.840788 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.840765 2566 scope.go:117] "RemoveContainer" containerID="dfdbd4fa0559ceda17f8e1e3a4de1f4df9acae759c3933199e028dbb75392558" Jul 17 22:24:10.840922 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.840886 2566 scope.go:117] "RemoveContainer" containerID="0b1a9ee9a6a03c6dd27edc75b474a6e8a84a06855c35e30b149721c730da7484" Jul 17 22:24:10.841172 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:10.841151 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6_odh-ai-gateway-infra(7e8612b4-7c51-4322-9ae8-f0f53d2dd480)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" Jul 17 22:24:10.879684 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.879648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a1cabb-4f72-4246-97ae-d5b23b10e394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq\" (UID: \"a7a1cabb-4f72-4246-97ae-d5b23b10e394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" Jul 17 22:24:10.880325 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.880306 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a1cabb-4f72-4246-97ae-d5b23b10e394-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq\" (UID: \"a7a1cabb-4f72-4246-97ae-d5b23b10e394\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" Jul 17 22:24:10.946705 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:10.946673 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" Jul 17 22:24:11.080918 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:11.080890 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq"] Jul 17 22:24:11.082825 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:24:11.082799 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a1cabb_4f72_4246_97ae_d5b23b10e394.slice/crio-2b01e82b7605fae78a536afdaab0833e9e93ec95bafa52a26878fa826530370f WatchSource:0}: Error finding container 2b01e82b7605fae78a536afdaab0833e9e93ec95bafa52a26878fa826530370f: Status 404 returned error can't find the container with id 2b01e82b7605fae78a536afdaab0833e9e93ec95bafa52a26878fa826530370f Jul 17 22:24:11.116163 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:11.116139 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6_7e8612b4-7c51-4322-9ae8-f0f53d2dd480/revoke-keys/1.log" Jul 17 22:24:11.847152 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:11.847118 2566 scope.go:117] "RemoveContainer" containerID="0b1a9ee9a6a03c6dd27edc75b474a6e8a84a06855c35e30b149721c730da7484" Jul 17 22:24:11.847702 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:11.847417 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6_odh-ai-gateway-infra(7e8612b4-7c51-4322-9ae8-f0f53d2dd480)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" Jul 17 22:24:11.848533 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:11.848500 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerID="58e47641c0c2682961783436e20aff683990948f8933cfab4d3611307215dee9" exitCode=6 Jul 17 22:24:11.848591 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:11.848542 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" event={"ID":"a7a1cabb-4f72-4246-97ae-d5b23b10e394","Type":"ContainerDied","Data":"58e47641c0c2682961783436e20aff683990948f8933cfab4d3611307215dee9"} Jul 17 22:24:11.848591 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:11.848576 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" event={"ID":"a7a1cabb-4f72-4246-97ae-d5b23b10e394","Type":"ContainerStarted","Data":"2b01e82b7605fae78a536afdaab0833e9e93ec95bafa52a26878fa826530370f"} Jul 17 22:24:11.848744 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:11.848730 2566 scope.go:117] "RemoveContainer" containerID="58e47641c0c2682961783436e20aff683990948f8933cfab4d3611307215dee9" Jul 17 22:24:12.855344 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:12.855219 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerID="550e7057ddf8665206d974a38bb2f21f9e86a2ed0ec9043b74d64a213d86f6ca" exitCode=6 Jul 17 22:24:12.855344 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:12.855307 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" event={"ID":"a7a1cabb-4f72-4246-97ae-d5b23b10e394","Type":"ContainerDied","Data":"550e7057ddf8665206d974a38bb2f21f9e86a2ed0ec9043b74d64a213d86f6ca"} Jul 17 22:24:12.855830 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:12.855379 2566 scope.go:117] "RemoveContainer" containerID="58e47641c0c2682961783436e20aff683990948f8933cfab4d3611307215dee9" Jul 17 22:24:12.855830 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:12.855561 2566 scope.go:117] "RemoveContainer" containerID="550e7057ddf8665206d974a38bb2f21f9e86a2ed0ec9043b74d64a213d86f6ca" Jul 17 22:24:12.855830 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:12.855801 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq_odh-ai-gateway-infra(a7a1cabb-4f72-4246-97ae-d5b23b10e394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" Jul 17 22:24:13.046063 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:13.046035 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mq5jp_c9f38acd-80dd-4365-9cfa-05cc86f18150/manager/0.log" Jul 17 22:24:13.155281 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:13.155173 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pdnfd_c593492f-d381-4054-9845-794427bd2bde/manager/0.log" Jul 17 22:24:13.374079 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:13.374044 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-njq74_8743d429-dd45-4cac-a688-8e976e5fd43e/registry-server/0.log" Jul 17 22:24:13.487982 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:13.487952 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5kz77_ece64894-51b5-488d-8fb3-7864be73d0d0/manager/0.log" Jul 17 22:24:13.600174 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:13.600147 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jxrtm_4081ac11-e146-4bbb-b6d0-aee0f5826429/limitador/0.log" Jul 17 22:24:13.861690 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:13.861604 2566 scope.go:117] "RemoveContainer" containerID="550e7057ddf8665206d974a38bb2f21f9e86a2ed0ec9043b74d64a213d86f6ca" Jul 17 22:24:13.862078 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:13.861810 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq_odh-ai-gateway-infra(a7a1cabb-4f72-4246-97ae-d5b23b10e394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" Jul 17 22:24:14.052991 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.052956 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p"] Jul 17 22:24:14.057790 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.057770 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" Jul 17 22:24:14.063776 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.063743 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p"] Jul 17 22:24:14.076272 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.076218 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678wmx4d_7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9/istio-proxy/0.log" Jul 17 22:24:14.194242 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.194196 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cv98c_b96ade98-d3a6-41e7-b78b-3f50d11f9df6/discovery/0.log" Jul 17 22:24:14.210914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.210878 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ff2a57-6960-4365-8216-a510f55253f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p\" (UID: \"37ff2a57-6960-4365-8216-a510f55253f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" Jul 17 22:24:14.303965 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.303936 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-5mrmr_0d903116-17f2-4664-8e0e-6499258fab01/kube-auth-proxy/0.log" Jul 17 22:24:14.312205 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.312173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ff2a57-6960-4365-8216-a510f55253f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p\" (UID: \"37ff2a57-6960-4365-8216-a510f55253f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" Jul 17 22:24:14.312766 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.312747 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ff2a57-6960-4365-8216-a510f55253f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p\" (UID: \"37ff2a57-6960-4365-8216-a510f55253f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" Jul 17 22:24:14.372287 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.372254 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" Jul 17 22:24:14.508873 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.508735 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p"] Jul 17 22:24:14.511465 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:24:14.511433 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ff2a57_6960_4365_8216_a510f55253f1.slice/crio-82760b895c610193540a527f5b6fd96157f44d0b151fefd6f2fe83419f91741b WatchSource:0}: Error finding container 82760b895c610193540a527f5b6fd96157f44d0b151fefd6f2fe83419f91741b: Status 404 returned error can't find the container with id 82760b895c610193540a527f5b6fd96157f44d0b151fefd6f2fe83419f91741b Jul 17 22:24:14.537047 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.537025 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-zzvnp_6eb8db5f-eacd-4224-9371-01f89f6ad86c/istio-proxy/1.log" Jul 17 22:24:14.867743 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.867713 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" event={"ID":"37ff2a57-6960-4365-8216-a510f55253f1","Type":"ContainerStarted","Data":"0c4e6b5c663c1c0c64b32eecce9dd2c69ae0f751c0b6909070a286f38d78ad77"} Jul 17 22:24:14.868146 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.867768 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" event={"ID":"37ff2a57-6960-4365-8216-a510f55253f1","Type":"ContainerStarted","Data":"82760b895c610193540a527f5b6fd96157f44d0b151fefd6f2fe83419f91741b"} Jul 17 22:24:14.882509 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:14.882461 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" podStartSLOduration=0.882444557 podStartE2EDuration="882.444557ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:14.881071757 +0000 UTC m=+3392.151069460" watchObservedRunningTime="2026-07-17 22:24:14.882444557 +0000 UTC m=+3392.152442261" Jul 17 22:24:15.708381 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:15.708354 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ht9s7_a2870c21-fe65-405c-a686-238eee98e265/main/0.log" Jul 17 22:24:15.874279 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:15.874220 2566 generic.go:356] "Generic (PLEG): container finished" podID="37ff2a57-6960-4365-8216-a510f55253f1" containerID="0c4e6b5c663c1c0c64b32eecce9dd2c69ae0f751c0b6909070a286f38d78ad77" exitCode=6 Jul 17 22:24:15.874279 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:15.874271 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" event={"ID":"37ff2a57-6960-4365-8216-a510f55253f1","Type":"ContainerDied","Data":"0c4e6b5c663c1c0c64b32eecce9dd2c69ae0f751c0b6909070a286f38d78ad77"} Jul 17 22:24:15.874711 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:15.874522 2566 scope.go:117] "RemoveContainer" containerID="0c4e6b5c663c1c0c64b32eecce9dd2c69ae0f751c0b6909070a286f38d78ad77" Jul 17 22:24:16.882090 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:16.882001 2566 generic.go:356] "Generic (PLEG): container finished" podID="37ff2a57-6960-4365-8216-a510f55253f1" containerID="6c514ae0b5679eb4714d1805db233a9f3dc6ba575707de2bb1e2b166d67adffc" exitCode=6 Jul 17 22:24:16.882510 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:16.882099 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" event={"ID":"37ff2a57-6960-4365-8216-a510f55253f1","Type":"ContainerDied","Data":"6c514ae0b5679eb4714d1805db233a9f3dc6ba575707de2bb1e2b166d67adffc"} Jul 17 22:24:16.882510 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:16.882158 2566 scope.go:117] "RemoveContainer" containerID="0c4e6b5c663c1c0c64b32eecce9dd2c69ae0f751c0b6909070a286f38d78ad77" Jul 17 22:24:16.882510 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:16.882415 2566 scope.go:117] "RemoveContainer" containerID="6c514ae0b5679eb4714d1805db233a9f3dc6ba575707de2bb1e2b166d67adffc" Jul 17 22:24:16.882636 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:16.882620 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p_odh-ai-gateway-infra(37ff2a57-6960-4365-8216-a510f55253f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" podUID="37ff2a57-6960-4365-8216-a510f55253f1" Jul 17 22:24:17.888776 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:17.888750 2566 scope.go:117] "RemoveContainer" containerID="6c514ae0b5679eb4714d1805db233a9f3dc6ba575707de2bb1e2b166d67adffc" Jul 17 22:24:17.889145 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:17.888957 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p_odh-ai-gateway-infra(37ff2a57-6960-4365-8216-a510f55253f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" podUID="37ff2a57-6960-4365-8216-a510f55253f1" Jul 17 22:24:24.716852 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:24.716816 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xbgxj_4f6911d6-a498-4c63-bb37-156b93e3a189/global-pull-secret-syncer/0.log" Jul 17 22:24:24.795633 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:24.795603 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gjgzw_780726fe-2e6b-4033-99a1-bf74fd25d7c7/konnectivity-agent/0.log" Jul 17 22:24:24.903709 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:24.903684 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-178.ec2.internal_6d7ce4753063ea67abc0aa6d62b0262b/haproxy/0.log" Jul 17 22:24:25.292867 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:25.292833 2566 scope.go:117] "RemoveContainer" containerID="0b1a9ee9a6a03c6dd27edc75b474a6e8a84a06855c35e30b149721c730da7484" Jul 17 22:24:25.929687 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:25.929657 2566 generic.go:356] "Generic (PLEG): container finished" podID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerID="4b1cee5817080d092878beab11733f424d57e5b7eaf85388706d200566272f34" exitCode=6 Jul 17 22:24:25.930081 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:25.929733 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" event={"ID":"7e8612b4-7c51-4322-9ae8-f0f53d2dd480","Type":"ContainerDied","Data":"4b1cee5817080d092878beab11733f424d57e5b7eaf85388706d200566272f34"} Jul 17 22:24:25.930081 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:25.929798 2566 scope.go:117] "RemoveContainer" containerID="0b1a9ee9a6a03c6dd27edc75b474a6e8a84a06855c35e30b149721c730da7484" Jul 17 22:24:25.930081 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:25.930063 2566 scope.go:117] "RemoveContainer" containerID="4b1cee5817080d092878beab11733f424d57e5b7eaf85388706d200566272f34" Jul 17 22:24:25.930325 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:25.930303 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6_odh-ai-gateway-infra(7e8612b4-7c51-4322-9ae8-f0f53d2dd480)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" Jul 17 22:24:26.954987 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:26.954954 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6"] Jul 17 22:24:27.095787 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.095760 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" Jul 17 22:24:27.121521 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.121491 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8612b4-7c51-4322-9ae8-f0f53d2dd480-maas-api-service-ca\") pod \"7e8612b4-7c51-4322-9ae8-f0f53d2dd480\" (UID: \"7e8612b4-7c51-4322-9ae8-f0f53d2dd480\") " Jul 17 22:24:27.121840 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.121816 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8612b4-7c51-4322-9ae8-f0f53d2dd480-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e8612b4-7c51-4322-9ae8-f0f53d2dd480" (UID: "7e8612b4-7c51-4322-9ae8-f0f53d2dd480"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:24:27.222135 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.222047 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8612b4-7c51-4322-9ae8-f0f53d2dd480-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:27.943117 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.943088 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" Jul 17 22:24:27.943339 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.943124 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6" event={"ID":"7e8612b4-7c51-4322-9ae8-f0f53d2dd480","Type":"ContainerDied","Data":"844ca605f5172c3e2944d2695dced00911fd2ea0d273fadbb884ea7c3b0cf31e"} Jul 17 22:24:27.943339 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.943198 2566 scope.go:117] "RemoveContainer" containerID="4b1cee5817080d092878beab11733f424d57e5b7eaf85388706d200566272f34" Jul 17 22:24:27.958922 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.958899 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6"] Jul 17 22:24:27.962523 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:27.962501 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-281d4ac2-2dzq6"] Jul 17 22:24:28.292732 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:28.292645 2566 scope.go:117] "RemoveContainer" containerID="550e7057ddf8665206d974a38bb2f21f9e86a2ed0ec9043b74d64a213d86f6ca" Jul 17 22:24:28.950843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:28.950802 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerID="ffdf70227ca7e00c47a56f480687f6c610d85f146c035c18c058cdf9e1a08233" exitCode=6 Jul 17 22:24:28.951034 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:28.950863 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" event={"ID":"a7a1cabb-4f72-4246-97ae-d5b23b10e394","Type":"ContainerDied","Data":"ffdf70227ca7e00c47a56f480687f6c610d85f146c035c18c058cdf9e1a08233"} Jul 17 22:24:28.951034 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:28.950921 2566 scope.go:117] "RemoveContainer" containerID="550e7057ddf8665206d974a38bb2f21f9e86a2ed0ec9043b74d64a213d86f6ca" Jul 17 22:24:28.951265 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:28.951220 2566 scope.go:117] "RemoveContainer" containerID="ffdf70227ca7e00c47a56f480687f6c610d85f146c035c18c058cdf9e1a08233" Jul 17 22:24:28.951488 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:28.951463 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq_odh-ai-gateway-infra(a7a1cabb-4f72-4246-97ae-d5b23b10e394)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" Jul 17 22:24:29.298062 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:29.298031 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" path="/var/lib/kubelet/pods/7e8612b4-7c51-4322-9ae8-f0f53d2dd480/volumes" Jul 17 22:24:29.974218 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:29.974182 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq"] Jul 17 22:24:30.121839 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.121813 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" Jul 17 22:24:30.148037 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.147992 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a1cabb-4f72-4246-97ae-d5b23b10e394-maas-api-service-ca\") pod \"a7a1cabb-4f72-4246-97ae-d5b23b10e394\" (UID: \"a7a1cabb-4f72-4246-97ae-d5b23b10e394\") " Jul 17 22:24:30.149294 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.149263 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a1cabb-4f72-4246-97ae-d5b23b10e394-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7a1cabb-4f72-4246-97ae-d5b23b10e394" (UID: "a7a1cabb-4f72-4246-97ae-d5b23b10e394"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:24:30.156113 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.156088 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mq5jp_c9f38acd-80dd-4365-9cfa-05cc86f18150/manager/0.log" Jul 17 22:24:30.179846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.179813 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-pdnfd_c593492f-d381-4054-9845-794427bd2bde/manager/0.log" Jul 17 22:24:30.248933 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.248852 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7a1cabb-4f72-4246-97ae-d5b23b10e394-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:30.264962 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.264929 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-njq74_8743d429-dd45-4cac-a688-8e976e5fd43e/registry-server/0.log" Jul 17 22:24:30.379457 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.379422 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5kz77_ece64894-51b5-488d-8fb3-7864be73d0d0/manager/0.log" Jul 17 22:24:30.398305 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.398267 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jxrtm_4081ac11-e146-4bbb-b6d0-aee0f5826429/limitador/0.log" Jul 17 22:24:30.964682 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.964642 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" event={"ID":"a7a1cabb-4f72-4246-97ae-d5b23b10e394","Type":"ContainerDied","Data":"2b01e82b7605fae78a536afdaab0833e9e93ec95bafa52a26878fa826530370f"} Jul 17 22:24:30.964682 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.964665 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq" Jul 17 22:24:30.964926 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.964718 2566 scope.go:117] "RemoveContainer" containerID="ffdf70227ca7e00c47a56f480687f6c610d85f146c035c18c058cdf9e1a08233" Jul 17 22:24:30.989325 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.988394 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq"] Jul 17 22:24:30.995758 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:30.995728 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9bc6369-r2cfq"] Jul 17 22:24:31.293445 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:31.293354 2566 scope.go:117] "RemoveContainer" containerID="6c514ae0b5679eb4714d1805db233a9f3dc6ba575707de2bb1e2b166d67adffc" Jul 17 22:24:31.298738 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:31.298712 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" path="/var/lib/kubelet/pods/a7a1cabb-4f72-4246-97ae-d5b23b10e394/volumes" Jul 17 22:24:31.972892 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:31.972804 2566 generic.go:356] "Generic (PLEG): container finished" podID="37ff2a57-6960-4365-8216-a510f55253f1" containerID="27cee9cc930739c1183dd97e32a3029d08d2a232f18fda876622ffced1fa5863" exitCode=6 Jul 17 22:24:31.972892 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:31.972861 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" event={"ID":"37ff2a57-6960-4365-8216-a510f55253f1","Type":"ContainerDied","Data":"27cee9cc930739c1183dd97e32a3029d08d2a232f18fda876622ffced1fa5863"} Jul 17 22:24:31.973430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:31.972909 2566 scope.go:117] "RemoveContainer" containerID="6c514ae0b5679eb4714d1805db233a9f3dc6ba575707de2bb1e2b166d67adffc" Jul 17 22:24:31.973430 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:31.973330 2566 scope.go:117] "RemoveContainer" containerID="27cee9cc930739c1183dd97e32a3029d08d2a232f18fda876622ffced1fa5863" Jul 17 22:24:31.973598 ip-10-0-139-178 kubenswrapper[2566]: E0717 22:24:31.973576 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p_odh-ai-gateway-infra(37ff2a57-6960-4365-8216-a510f55253f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" podUID="37ff2a57-6960-4365-8216-a510f55253f1" Jul 17 22:24:32.293479 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.293406 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-knzmh_1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d/kube-state-metrics/0.log" Jul 17 22:24:32.315583 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.315553 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-knzmh_1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d/kube-rbac-proxy-main/0.log" Jul 17 22:24:32.339049 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.339021 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-knzmh_1a8d31bd-ebd4-4f0a-a394-b6e22fba3e4d/kube-rbac-proxy-self/0.log" Jul 17 22:24:32.363437 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.363392 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-589fb9d4f5-hcd2p_bcf151ce-e42d-47f3-a732-d982b179c7e7/metrics-server/0.log" Jul 17 22:24:32.387543 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.387516 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-qzsz6_cd61feb2-effd-4c91-a9af-2c04eea7a815/monitoring-plugin/0.log" Jul 17 22:24:32.415046 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.415020 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-72t6s_144d3e2b-9352-47e1-bb25-434bb2266554/node-exporter/0.log" Jul 17 22:24:32.439969 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.439944 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-72t6s_144d3e2b-9352-47e1-bb25-434bb2266554/kube-rbac-proxy/0.log" Jul 17 22:24:32.465072 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.465047 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-72t6s_144d3e2b-9352-47e1-bb25-434bb2266554/init-textfile/0.log" Jul 17 22:24:32.751996 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.751963 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a14f11d-0671-427b-a932-c69238e6f818/prometheus/0.log" Jul 17 22:24:32.774330 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.774301 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a14f11d-0671-427b-a932-c69238e6f818/config-reloader/0.log" Jul 17 22:24:32.795116 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.795087 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a14f11d-0671-427b-a932-c69238e6f818/thanos-sidecar/0.log" Jul 17 22:24:32.816006 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.815975 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a14f11d-0671-427b-a932-c69238e6f818/kube-rbac-proxy-web/0.log" Jul 17 22:24:32.836478 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.836459 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a14f11d-0671-427b-a932-c69238e6f818/kube-rbac-proxy/0.log" Jul 17 22:24:32.858303 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.858279 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a14f11d-0671-427b-a932-c69238e6f818/kube-rbac-proxy-thanos/0.log" Jul 17 22:24:32.878013 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.877987 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2a14f11d-0671-427b-a932-c69238e6f818/init-config-reloader/0.log" Jul 17 22:24:32.905867 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.905840 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-c2l8k_749c3879-6357-483a-8640-795e87d62b2c/prometheus-operator/0.log" Jul 17 22:24:32.925289 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.925263 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-c2l8k_749c3879-6357-483a-8640-795e87d62b2c/kube-rbac-proxy/0.log" Jul 17 22:24:32.954139 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:32.954105 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-lxrb8_0a4ceea4-d7ad-428d-b9f9-ea2fe6155094/prometheus-operator-admission-webhook/0.log" Jul 17 22:24:33.000359 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.000324 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p"] Jul 17 22:24:33.143108 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.143073 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" Jul 17 22:24:33.175121 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.175066 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ff2a57-6960-4365-8216-a510f55253f1-maas-api-service-ca\") pod \"37ff2a57-6960-4365-8216-a510f55253f1\" (UID: \"37ff2a57-6960-4365-8216-a510f55253f1\") " Jul 17 22:24:33.175411 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.175390 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ff2a57-6960-4365-8216-a510f55253f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37ff2a57-6960-4365-8216-a510f55253f1" (UID: "37ff2a57-6960-4365-8216-a510f55253f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:24:33.275843 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.275796 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ff2a57-6960-4365-8216-a510f55253f1-maas-api-service-ca\") on node \"ip-10-0-139-178.ec2.internal\" DevicePath \"\"" Jul 17 22:24:33.985092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.985018 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" Jul 17 22:24:33.985092 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.985057 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p" event={"ID":"37ff2a57-6960-4365-8216-a510f55253f1","Type":"ContainerDied","Data":"82760b895c610193540a527f5b6fd96157f44d0b151fefd6f2fe83419f91741b"} Jul 17 22:24:33.985335 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:33.985151 2566 scope.go:117] "RemoveContainer" containerID="27cee9cc930739c1183dd97e32a3029d08d2a232f18fda876622ffced1fa5863" Jul 17 22:24:34.000546 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:34.000518 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p"] Jul 17 22:24:34.004018 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:34.003995 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f4935d4-phw6p"] Jul 17 22:24:35.017207 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017172 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj"] Jul 17 22:24:35.017719 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017699 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ff2a57-6960-4365-8216-a510f55253f1" containerName="revoke-keys" Jul 17 22:24:35.017807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017722 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ff2a57-6960-4365-8216-a510f55253f1" containerName="revoke-keys" Jul 17 22:24:35.017807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017737 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.017807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017746 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.017807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017761 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.017807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017770 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.017807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017790 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ff2a57-6960-4365-8216-a510f55253f1" containerName="revoke-keys" Jul 17 22:24:35.017807 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017799 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ff2a57-6960-4365-8216-a510f55253f1" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017813 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017821 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017834 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017842 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017856 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017864 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017874 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017881 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017970 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017982 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.017991 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e8612b4-7c51-4322-9ae8-f0f53d2dd480" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.018000 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a1cabb-4f72-4246-97ae-d5b23b10e394" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.018009 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ff2a57-6960-4365-8216-a510f55253f1" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.018017 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ff2a57-6960-4365-8216-a510f55253f1" containerName="revoke-keys" Jul 17 22:24:35.018143 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.018029 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ff2a57-6960-4365-8216-a510f55253f1" containerName="revoke-keys" Jul 17 22:24:35.022535 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.022511 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.025087 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.025066 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qg69\"/\"openshift-service-ca.crt\"" Jul 17 22:24:35.025087 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.025085 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5qg69\"/\"kube-root-ca.crt\"" Jul 17 22:24:35.026098 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.026078 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5qg69\"/\"default-dockercfg-ljcb8\"" Jul 17 22:24:35.029792 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.029764 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj"] Jul 17 22:24:35.090186 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.090145 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-proc\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.090392 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.090197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskg9\" (UniqueName: \"kubernetes.io/projected/e84abe5f-90c1-419b-9a73-90551e6c023e-kube-api-access-lskg9\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.090392 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.090254 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-lib-modules\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.090392 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.090343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-podres\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.090392 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.090377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-sys\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.191846 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.191800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-lib-modules\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192021 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.191871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-podres\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192021 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.191900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-sys\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192021 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.191951 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-proc\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192021 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.192013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskg9\" (UniqueName: \"kubernetes.io/projected/e84abe5f-90c1-419b-9a73-90551e6c023e-kube-api-access-lskg9\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192155 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.192026 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-lib-modules\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192155 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.192035 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-podres\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192155 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.192028 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-sys\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.192155 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.192067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e84abe5f-90c1-419b-9a73-90551e6c023e-proc\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.200556 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.200534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskg9\" (UniqueName: \"kubernetes.io/projected/e84abe5f-90c1-419b-9a73-90551e6c023e-kube-api-access-lskg9\") pod \"perf-node-gather-daemonset-n2hzj\" (UID: \"e84abe5f-90c1-419b-9a73-90551e6c023e\") " pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.299036 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.298953 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ff2a57-6960-4365-8216-a510f55253f1" path="/var/lib/kubelet/pods/37ff2a57-6960-4365-8216-a510f55253f1/volumes" Jul 17 22:24:35.334219 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.334190 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:35.472309 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.472283 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj"] Jul 17 22:24:35.476778 ip-10-0-139-178 kubenswrapper[2566]: W0717 22:24:35.474500 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode84abe5f_90c1_419b_9a73_90551e6c023e.slice/crio-e8c55d3c945a2118b04d07df2bd38662d17d755b1b78e1a8258bcf0e4a6a6b2f WatchSource:0}: Error finding container e8c55d3c945a2118b04d07df2bd38662d17d755b1b78e1a8258bcf0e4a6a6b2f: Status 404 returned error can't find the container with id e8c55d3c945a2118b04d07df2bd38662d17d755b1b78e1a8258bcf0e4a6a6b2f Jul 17 22:24:35.997966 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.997925 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" event={"ID":"e84abe5f-90c1-419b-9a73-90551e6c023e","Type":"ContainerStarted","Data":"3c4e811a11ac9750bc108d90a38484a8ee435f10e3eede179a98c89c196f6e75"} Jul 17 22:24:35.998126 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.997984 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" event={"ID":"e84abe5f-90c1-419b-9a73-90551e6c023e","Type":"ContainerStarted","Data":"e8c55d3c945a2118b04d07df2bd38662d17d755b1b78e1a8258bcf0e4a6a6b2f"} Jul 17 22:24:35.998126 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:35.998011 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:36.015343 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:36.015289 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" podStartSLOduration=2.015272681 podStartE2EDuration="2.015272681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:24:36.013303405 +0000 UTC m=+3413.283301125" watchObservedRunningTime="2026-07-17 22:24:36.015272681 +0000 UTC m=+3413.285270384" Jul 17 22:24:37.334884 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:37.334856 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-df5k2_ad2a2ed7-2f27-42ea-9516-f224e975dcd2/dns/0.log" Jul 17 22:24:37.358075 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:37.358046 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-df5k2_ad2a2ed7-2f27-42ea-9516-f224e975dcd2/kube-rbac-proxy/0.log" Jul 17 22:24:37.449571 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:37.449547 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5zrcf_1c44a901-fbcb-40f0-9984-123b95d7dbb5/dns-node-resolver/0.log" Jul 17 22:24:38.014095 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:38.014065 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-52cmr_45e3a6c9-bb86-424c-8909-feafef42373b/node-ca/0.log" Jul 17 22:24:39.004243 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:39.004204 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678wmx4d_7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9/istio-proxy/0.log" Jul 17 22:24:39.116534 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:39.116501 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cv98c_b96ade98-d3a6-41e7-b78b-3f50d11f9df6/discovery/0.log" Jul 17 22:24:39.137729 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:39.137703 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-5mrmr_0d903116-17f2-4664-8e0e-6499258fab01/kube-auth-proxy/0.log" Jul 17 22:24:39.198117 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:39.198069 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-zzvnp_6eb8db5f-eacd-4224-9371-01f89f6ad86c/istio-proxy/0.log" Jul 17 22:24:39.227533 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:39.227502 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-zzvnp_6eb8db5f-eacd-4224-9371-01f89f6ad86c/istio-proxy/1.log" Jul 17 22:24:40.033793 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:40.033761 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h2pz6_4aa80dd6-b323-48a1-b800-36603b5acaca/serve-healthcheck-canary/0.log" Jul 17 22:24:40.829215 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:40.829183 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7hk_00f31afe-10c6-46c9-bed8-79a188a026da/kube-rbac-proxy/0.log" Jul 17 22:24:40.848159 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:40.848129 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7hk_00f31afe-10c6-46c9-bed8-79a188a026da/exporter/0.log" Jul 17 22:24:40.868655 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:40.868629 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7hk_00f31afe-10c6-46c9-bed8-79a188a026da/extractor/0.log" Jul 17 22:24:42.013478 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:42.013442 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5qg69/perf-node-gather-daemonset-n2hzj" Jul 17 22:24:43.065155 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:43.065124 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-m5p9g_e9da3247-ff20-4211-ad5f-640426eb1a6d/manager/0.log" Jul 17 22:24:43.525914 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:43.525884 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-v2w56_48cf6d79-d7e0-451b-a3c6-53f54049e3b3/manager/0.log" Jul 17 22:24:43.545933 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:43.545909 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-964b768d-v2w56_48cf6d79-d7e0-451b-a3c6-53f54049e3b3/copy-manifests/0.log" Jul 17 22:24:45.017282 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:45.017248 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ck9vn_4d6b63ad-6de7-4e09-8634-6d9d2bd0f289/openshift-lws-operator/0.log" Jul 17 22:24:45.551375 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:45.551348 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678wmx4d_7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9/istio-proxy/0.log" Jul 17 22:24:45.655115 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:45.655088 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cv98c_b96ade98-d3a6-41e7-b78b-3f50d11f9df6/discovery/0.log" Jul 17 22:24:45.665174 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:45.665156 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-5mrmr_0d903116-17f2-4664-8e0e-6499258fab01/kube-auth-proxy/0.log" Jul 17 22:24:45.696877 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:45.696855 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-zzvnp_6eb8db5f-eacd-4224-9371-01f89f6ad86c/istio-proxy/0.log" Jul 17 22:24:45.726288 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:45.726253 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-zzvnp_6eb8db5f-eacd-4224-9371-01f89f6ad86c/istio-proxy/1.log" Jul 17 22:24:46.992072 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:46.992045 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678wmx4d_7d5f1fbc-c75c-4d00-a1ba-4f1aa073e3e9/istio-proxy/0.log" Jul 17 22:24:47.103529 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:47.103486 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cv98c_b96ade98-d3a6-41e7-b78b-3f50d11f9df6/discovery/0.log" Jul 17 22:24:47.115715 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:47.115688 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788f6fc9d-5mrmr_0d903116-17f2-4664-8e0e-6499258fab01/kube-auth-proxy/0.log" Jul 17 22:24:47.152170 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:47.152139 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-zzvnp_6eb8db5f-eacd-4224-9371-01f89f6ad86c/istio-proxy/0.log" Jul 17 22:24:47.182389 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:47.182358 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-zzvnp_6eb8db5f-eacd-4224-9371-01f89f6ad86c/istio-proxy/1.log" Jul 17 22:24:51.333831 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.333802 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kfh2f_302e487b-c564-42d3-90d7-a846a77e2490/kube-multus-additional-cni-plugins/0.log" Jul 17 22:24:51.353395 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.353371 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kfh2f_302e487b-c564-42d3-90d7-a846a77e2490/egress-router-binary-copy/0.log" Jul 17 22:24:51.372574 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.372545 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kfh2f_302e487b-c564-42d3-90d7-a846a77e2490/cni-plugins/0.log" Jul 17 22:24:51.391537 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.391508 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kfh2f_302e487b-c564-42d3-90d7-a846a77e2490/bond-cni-plugin/0.log" Jul 17 22:24:51.411384 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.411360 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kfh2f_302e487b-c564-42d3-90d7-a846a77e2490/routeoverride-cni/0.log" Jul 17 22:24:51.431871 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.431849 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kfh2f_302e487b-c564-42d3-90d7-a846a77e2490/whereabouts-cni-bincopy/0.log" Jul 17 22:24:51.452849 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.452818 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kfh2f_302e487b-c564-42d3-90d7-a846a77e2490/whereabouts-cni/0.log" Jul 17 22:24:51.741410 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.741378 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vphpc_546a56e5-7cfe-403b-8ca8-625a943fbfb1/kube-multus/0.log" Jul 17 22:24:51.809095 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.809066 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g4x5v_29753f77-c24d-42c7-8518-24fd9698d3ac/network-metrics-daemon/0.log" Jul 17 22:24:51.829482 ip-10-0-139-178 kubenswrapper[2566]: I0717 22:24:51.829451 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g4x5v_29753f77-c24d-42c7-8518-24fd9698d3ac/kube-rbac-proxy/0.log"