Jul 21 08:16:49.717780 ip-10-0-131-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 08:16:49.717795 ip-10-0-131-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 08:16:49.717805 ip-10-0-131-134 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 08:16:49.718282 ip-10-0-131-134 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 08:16:59.838653 ip-10-0-131-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 08:16:59.838671 ip-10-0-131-134 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3d599b05a26e4fa8b68d41339ae2982f -- Jul 21 08:19:03.992630 ip-10-0-131-134 systemd[1]: Starting Kubernetes Kubelet... Jul 21 08:19:04.411699 ip-10-0-131-134 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 08:19:04.411699 ip-10-0-131-134 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 08:19:04.411699 ip-10-0-131-134 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 08:19:04.411699 ip-10-0-131-134 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 08:19:04.411699 ip-10-0-131-134 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 08:19:04.413607 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.413455 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 08:19:04.419471 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419452 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419473 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419477 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419480 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419483 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419486 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419488 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419491 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419494 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419496 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419499 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419501 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419504 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419506 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419510 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419513 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419516 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419526 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:19:04.419520 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419529 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419532 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419535 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419537 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419540 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419543 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419546 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419548 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419551 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419554 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419556 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419558 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419561 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419563 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419566 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419568 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419571 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419573 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419576 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419578 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:19:04.419952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419581 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419584 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419587 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419589 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419591 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419594 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419596 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419598 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419601 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419603 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419606 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419608 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419610 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419613 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419616 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419621 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419625 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419628 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419631 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:19:04.420484 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419634 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419637 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419640 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419643 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419645 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419648 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419650 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419653 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419655 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419658 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419661 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419663 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419666 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419668 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419671 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419675 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419679 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419683 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419686 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:19:04.420935 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419688 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419691 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419693 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419696 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419698 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419701 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419703 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419706 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419710 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419713 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419716 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419718 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419721 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419725 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419728 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419731 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419734 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419737 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419739 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419742 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:19:04.421425 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419744 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:19:04.421940 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419747 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:19:04.421940 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.419749 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:19:04.423413 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423391 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:19:04.423413 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423410 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:19:04.423413 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423414 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:19:04.423413 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423417 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:19:04.423413 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423420 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423422 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423426 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423429 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423432 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423435 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423438 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423440 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423443 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423446 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423449 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423451 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423454 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423456 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423459 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423461 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423464 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423466 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423469 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423471 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:19:04.423605 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423474 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423477 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423479 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423482 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423485 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423487 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423490 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423493 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423495 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423497 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423500 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423503 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423506 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423508 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423510 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423513 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423515 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423517 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423520 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423523 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:19:04.424102 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423525 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423528 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423530 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423532 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423535 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423537 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423540 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423542 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423545 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423548 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423550 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423552 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423555 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423558 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423561 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423563 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423566 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423568 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423571 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423574 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:19:04.424638 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423576 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423578 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423581 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423583 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423586 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423588 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423591 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423593 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423598 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423601 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423604 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423609 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423613 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423616 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423619 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423621 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423624 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423627 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423629 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:19:04.425131 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423632 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423635 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423637 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423640 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423642 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423645 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423648 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423651 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423653 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423655 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423658 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423660 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423664 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423668 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423671 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.423674 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424119 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424125 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424128 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:19:04.425630 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424131 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424134 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424137 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424140 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424142 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424145 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424147 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424150 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424152 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424155 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424157 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424159 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424162 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424165 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424168 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424171 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424173 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424176 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424179 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424182 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:19:04.426100 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424185 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424187 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424190 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424192 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424194 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424197 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424199 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424202 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424204 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424207 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424210 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424213 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424215 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424218 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424221 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424224 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424228 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424230 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424233 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:19:04.426614 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424235 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424238 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424241 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424244 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424246 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424249 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424251 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424254 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424256 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424259 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424261 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424264 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424267 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424270 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424272 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424275 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424277 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424280 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424282 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424285 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:19:04.427074 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424287 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424290 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424292 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424296 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424300 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424303 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424323 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424326 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424329 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424331 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424334 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424336 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424339 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424343 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424346 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424348 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424352 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424355 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424358 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:19:04.427571 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424361 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424366 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424369 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424372 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424374 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424377 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424379 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424382 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424385 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424387 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424390 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424392 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424395 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424397 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424399 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424402 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424404 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.424407 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424475 2553 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424482 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424489 2553 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 08:19:04.428026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424495 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424501 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424506 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424513 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424519 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424523 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424526 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424531 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424534 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424537 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424540 2553 flags.go:64] FLAG: --cgroup-root="" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424543 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424546 2553 flags.go:64] FLAG: --client-ca-file="" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424550 2553 flags.go:64] FLAG: --cloud-provider="external" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424553 2553 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424559 2553 flags.go:64] FLAG: --cluster-domain="" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424562 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424566 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424569 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424572 2553 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424575 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424578 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424581 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 08:19:04.428600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424585 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424587 2553 flags.go:64] FLAG: --contention-profiling="false" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424590 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424593 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424597 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424600 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424604 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424607 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424609 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424612 2553 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424615 2553 flags.go:64] FLAG: --enable-server="true" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424618 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424622 2553 flags.go:64] FLAG: --event-burst="100" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424625 2553 flags.go:64] FLAG: --event-qps="50" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424628 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424631 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424634 2553 flags.go:64] FLAG: --eviction-hard="" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424637 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424640 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424643 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424646 2553 flags.go:64] FLAG: --eviction-soft="" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424649 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424652 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424656 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424659 2553 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 08:19:04.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424662 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424665 2553 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424667 2553 flags.go:64] FLAG: --feature-gates="" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424671 2553 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424674 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424677 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424680 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424683 2553 flags.go:64] FLAG: --healthz-port="10248" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424686 2553 flags.go:64] FLAG: --help="false" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424689 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424691 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424697 2553 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424700 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424703 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424706 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424709 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424712 2553 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424715 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424717 2553 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424720 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424723 2553 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424726 2553 flags.go:64] FLAG: --kube-reserved="" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424729 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424732 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 08:19:04.429920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424735 2553 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424737 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424740 2553 flags.go:64] FLAG: --lock-file="" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424743 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424746 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424749 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424754 2553 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424757 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424760 2553 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424763 2553 flags.go:64] FLAG: --logging-format="text" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424765 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424768 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424771 2553 flags.go:64] FLAG: --manifest-url="" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424774 2553 flags.go:64] FLAG: --manifest-url-header="" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424778 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424781 2553 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424784 2553 flags.go:64] FLAG: --max-pods="110" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424787 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424790 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424794 2553 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424796 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424799 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424802 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424805 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424813 2553 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 08:19:04.430523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424816 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424819 2553 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424822 2553 flags.go:64] FLAG: --pod-cidr="" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424824 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424830 2553 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424833 2553 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424836 2553 flags.go:64] FLAG: --pods-per-core="0" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424839 2553 flags.go:64] FLAG: --port="10250" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424842 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424845 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c3a82272ab4a2a74" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424848 2553 flags.go:64] FLAG: --qos-reserved="" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424851 2553 flags.go:64] FLAG: --read-only-port="10255" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424854 2553 flags.go:64] FLAG: --register-node="true" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424856 2553 flags.go:64] FLAG: --register-with-taints="" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424860 2553 flags.go:64] FLAG: --registry-burst="10" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424864 2553 flags.go:64] FLAG: --registry-qps="5" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424866 2553 flags.go:64] FLAG: --reserved-cpus="" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424869 2553 flags.go:64] FLAG: --reserved-memory="" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424873 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424875 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424878 2553 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424881 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424884 2553 flags.go:64] FLAG: --runonce="false" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424886 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424889 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 08:19:04.431140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424892 2553 flags.go:64] FLAG: --seccomp-default="false" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424895 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424899 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424901 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424904 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424907 2553 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424910 2553 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424913 2553 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424916 2553 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424918 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424921 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424924 2553 flags.go:64] FLAG: --system-cgroups="" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424927 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424932 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424935 2553 flags.go:64] FLAG: --tls-cert-file="" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424938 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424943 2553 flags.go:64] FLAG: --tls-min-version="" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424945 2553 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424948 2553 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424951 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424953 2553 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424956 2553 flags.go:64] FLAG: --v="2" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424960 2553 flags.go:64] FLAG: --version="false" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424965 2553 flags.go:64] FLAG: --vmodule="" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424969 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 08:19:04.431794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.424972 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425091 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425094 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425098 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425101 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425104 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425107 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425110 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425113 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425117 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425119 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425122 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425125 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425128 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425130 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425133 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425135 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425138 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425141 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425143 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:19:04.432429 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425146 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425149 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425151 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425154 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425156 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425159 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425161 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425163 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425166 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425169 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425173 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425177 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425181 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425184 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425187 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425189 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425192 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425194 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425197 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:19:04.432919 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425200 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425203 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425207 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425209 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425212 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425214 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425216 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425219 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425221 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425224 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425226 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425228 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425231 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425233 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425236 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425238 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425241 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425243 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425246 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425248 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:19:04.433405 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425250 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425253 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425255 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425258 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425260 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425263 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425266 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425268 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425270 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425273 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425276 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425278 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425282 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425284 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425288 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425291 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425302 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425317 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425320 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425323 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:19:04.433936 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425325 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425328 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425330 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425333 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425335 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425338 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425341 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425345 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425348 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425350 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425353 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425355 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425358 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425360 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425363 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425365 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425368 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425370 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425374 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:19:04.434436 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425377 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.425379 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.425996 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.432842 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.432859 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.432880 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.432886 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432945 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432951 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432955 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432958 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432961 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432964 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432967 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432970 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:19:04.434893 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432973 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432975 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432978 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432981 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432984 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432986 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432990 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432992 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432996 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.432998 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433001 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433003 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433006 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433008 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433011 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433013 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433016 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433018 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433021 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433024 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:19:04.435266 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433026 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433030 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433033 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433036 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433038 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433041 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433044 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433046 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433049 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433052 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433054 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433057 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433059 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433062 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433064 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433069 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433072 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433075 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433078 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:19:04.435764 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433080 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433083 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433085 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433087 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433090 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433092 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433095 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433097 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433100 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433103 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433107 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433110 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433113 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433115 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433118 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433122 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433125 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433128 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433130 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:19:04.436281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433133 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433135 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433138 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433141 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433143 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433147 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433150 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433153 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433156 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433159 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433161 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433164 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433166 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433169 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433171 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433174 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433176 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433178 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433181 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:19:04.436797 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433183 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433185 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433188 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433190 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433193 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433195 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433197 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433200 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433203 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433206 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433209 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433212 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433214 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433217 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.433222 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 08:19:04.437319 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433383 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433389 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433393 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433396 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433398 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433401 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433403 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433406 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433409 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433411 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433414 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433417 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433419 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433422 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433425 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433427 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433431 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433436 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433439 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433441 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 08:19:04.437720 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433444 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433447 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433450 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433452 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433455 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433458 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433461 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433464 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433467 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433469 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433472 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433474 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433476 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433479 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433481 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433484 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433487 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433489 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433491 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 08:19:04.438203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433494 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433497 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433499 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433501 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433504 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433506 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433509 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433511 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433514 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433516 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433518 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433520 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433523 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433525 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433527 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433530 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433532 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433534 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433538 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433540 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 08:19:04.438700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433543 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433545 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433550 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433553 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433556 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433559 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433562 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433564 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433567 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433569 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433572 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433574 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433576 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433578 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433581 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433585 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433588 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433591 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433593 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 08:19:04.439172 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433596 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433599 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433601 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433604 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433607 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433609 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433612 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433615 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433617 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433620 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433623 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433625 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433628 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433630 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433633 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433635 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433638 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433641 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433643 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433645 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 08:19:04.439640 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:04.433648 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 08:19:04.440113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.433652 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 08:19:04.440113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.433789 2553 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 08:19:04.440113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.436051 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 08:19:04.440113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.437478 2553 server.go:1019] "Starting client certificate rotation" Jul 21 08:19:04.440113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.437582 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 08:19:04.440113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.437619 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 08:19:04.440649 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.440634 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 08:19:04.441803 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.441783 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 08:19:04.452098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.452071 2553 log.go:25] "Validated CRI v1 runtime API" Jul 21 08:19:04.455690 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.455672 2553 log.go:25] "Validated CRI v1 image API" Jul 21 08:19:04.457414 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.457399 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 08:19:04.461106 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.461086 2553 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c9c415bd-b86e-4ae3-82d5-c9a777278afc:/dev/nvme0n1p3 e14e1fcf-9698-4262-9293-4b85c2182812:/dev/nvme0n1p4] Jul 21 08:19:04.461181 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.461105 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 08:19:04.465427 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.465291 2553 manager.go:216] Machine: {Timestamp:2026-07-21 08:19:04.464800478 +0000 UTC m=+0.350471018 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3061739 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28ebc862bd234cdd363e32b7c9ecef SystemUUID:ec28ebc8-62bd-234c-dd36-3e32b7c9ecef BootID:3d599b05-a26e-4fa8-b68d-41339ae2982f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:f7:6a:d3:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:f7:6a:d3:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:96:48:02:93:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 08:19:04.465427 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.465416 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 08:19:04.465553 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.465500 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 08:19:04.466677 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.466659 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 08:19:04.468637 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.468604 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 08:19:04.468808 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.468644 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 08:19:04.469456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.469445 2553 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 08:19:04.469517 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.469463 2553 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 08:19:04.469517 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.469474 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 08:19:04.469517 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.469490 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 08:19:04.470901 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.470889 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 08:19:04.474198 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474183 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 21 08:19:04.474366 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474354 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 08:19:04.474425 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474397 2553 kubelet.go:486] "Attempting to sync node with API server" Jul 21 08:19:04.474425 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474418 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 08:19:04.474515 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474438 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 08:19:04.474515 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474455 2553 kubelet.go:398] "Adding apiserver pod source" Jul 21 08:19:04.474515 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474467 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 08:19:04.474868 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474853 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 08:19:04.474955 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.474877 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 08:19:04.477099 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.477080 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 08:19:04.478962 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.478949 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 08:19:04.479022 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.478980 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 08:19:04.481518 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481502 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481522 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481529 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481535 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481544 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481550 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481558 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481564 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481572 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481578 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481586 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 08:19:04.481595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.481591 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 08:19:04.482450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.482439 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 08:19:04.482450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.482447 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 08:19:04.485462 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.485442 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f7zv" Jul 21 08:19:04.485677 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.485651 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 08:19:04.485757 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.485684 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 08:19:04.485858 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.485842 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 08:19:04.485948 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.485937 2553 server.go:1268] "Started kubelet" Jul 21 08:19:04.486051 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.486026 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 08:19:04.486180 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.486020 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 08:19:04.486281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.486199 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 08:19:04.486965 ip-10-0-131-134 systemd[1]: Started Kubernetes Kubelet. Jul 21 08:19:04.490438 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.490415 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 08:19:04.491566 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.491548 2553 server.go:310] "Adding debug handlers to kubelet server" Jul 21 08:19:04.491929 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.491908 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 08:19:04.492396 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.492372 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4f7zv" Jul 21 08:19:04.495484 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.494552 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-134.ec2.internal.18c4403403730df9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-134.ec2.internal,UID:ip-10-0-131-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-134.ec2.internal,},FirstTimestamp:2026-07-21 08:19:04.485907961 +0000 UTC m=+0.371578500,LastTimestamp:2026-07-21 08:19:04.485907961 +0000 UTC m=+0.371578500,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-134.ec2.internal,}" Jul 21 08:19:04.499437 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.499422 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 08:19:04.499522 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.499433 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 08:19:04.499574 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.499523 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:04.500128 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.500076 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 08:19:04.500205 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.500160 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 08:19:04.500205 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.500171 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 08:19:04.500473 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.500300 2553 reconstruct.go:97] "Volume reconstruction finished" Jul 21 08:19:04.500473 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.500324 2553 reconciler.go:29] "Reconciler: start to sync state" Jul 21 08:19:04.501526 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.501510 2553 factory.go:55] Registering systemd factory Jul 21 08:19:04.501526 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.501525 2553 factory.go:223] Registration of the systemd container factory successfully Jul 21 08:19:04.501775 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.501762 2553 factory.go:153] Registering CRI-O factory Jul 21 08:19:04.501834 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.501779 2553 factory.go:223] Registration of the crio container factory successfully Jul 21 08:19:04.501834 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.501831 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 08:19:04.501926 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.501855 2553 factory.go:103] Registering Raw factory Jul 21 08:19:04.501926 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.501871 2553 manager.go:1196] Started watching for new ooms in manager Jul 21 08:19:04.502491 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.502474 2553 manager.go:349] Starting recovery of all containers Jul 21 08:19:04.508000 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.507954 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:19:04.510630 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.510431 2553 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-134.ec2.internal\" not found" node="ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.513190 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.513174 2553 manager.go:354] Recovery completed Jul 21 08:19:04.518199 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.518186 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:19:04.520015 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.519996 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:19:04.520103 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.520031 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:19:04.520103 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.520043 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:19:04.520614 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.520598 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 08:19:04.520687 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.520619 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:19:04.520687 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.520612 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 08:19:04.520687 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.520642 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 21 08:19:04.522367 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.522352 2553 policy_none.go:49] "None policy: Start" Jul 21 08:19:04.522449 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.522373 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 08:19:04.522449 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.522388 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 08:19:04.522805 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.522786 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:19:04.522856 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.522826 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:19:04.522856 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.522839 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:19:04.524110 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.524097 2553 policy_none.go:47] "Start" Jul 21 08:19:04.554963 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.554943 2553 manager.go:337] "Starting Device Plugin manager" Jul 21 08:19:04.555085 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.554979 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 08:19:04.555085 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.554991 2553 server.go:87] "Starting device plugin registration server" Jul 21 08:19:04.555295 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.555284 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 08:19:04.555369 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.555299 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 08:19:04.555444 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.555427 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 08:19:04.555533 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.555520 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 08:19:04.555588 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.555535 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 08:19:04.557170 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.556986 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 08:19:04.557170 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.557042 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:04.655777 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.655735 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:19:04.656976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.656754 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:19:04.656976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.656787 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:19:04.656976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.656799 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:19:04.656976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.656832 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.662392 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.662350 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.662392 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.662373 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-134.ec2.internal\": node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:04.666226 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.666201 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 08:19:04.667505 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.667486 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 08:19:04.667614 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.667527 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 08:19:04.667614 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.667547 2553 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 08:19:04.667614 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.667576 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 08:19:04.669652 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.669580 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:19:04.676249 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.676231 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:04.767733 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.767676 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal"] Jul 21 08:19:04.767914 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.767767 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:19:04.769804 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.769780 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:19:04.769916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.769825 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:19:04.769916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.769840 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:19:04.771343 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.771329 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:19:04.771464 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.771448 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.771504 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.771489 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:19:04.772109 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.772095 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:19:04.772176 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.772122 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:19:04.772176 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.772131 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:19:04.772790 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.772773 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:19:04.772848 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.772803 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:19:04.772848 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.772814 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:19:04.774494 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.774478 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.774544 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.774512 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 08:19:04.775214 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.775202 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientMemory" Jul 21 08:19:04.775274 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.775231 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 08:19:04.775274 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.775243 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeHasSufficientPID" Jul 21 08:19:04.776646 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.776632 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:04.802615 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.802589 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/420479272364a70b28d1192701ae6b13-config\") pod \"kube-apiserver-proxy-ip-10-0-131-134.ec2.internal\" (UID: \"420479272364a70b28d1192701ae6b13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.802719 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.802619 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acacd1221e4ccabdaec949b2a26be37f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"acacd1221e4ccabdaec949b2a26be37f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.802719 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.802645 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acacd1221e4ccabdaec949b2a26be37f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"acacd1221e4ccabdaec949b2a26be37f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.809919 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.809902 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-134.ec2.internal\" not found" node="ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.814349 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.814332 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-134.ec2.internal\" not found" node="ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.877473 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.877435 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:04.902757 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.902730 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/420479272364a70b28d1192701ae6b13-config\") pod \"kube-apiserver-proxy-ip-10-0-131-134.ec2.internal\" (UID: \"420479272364a70b28d1192701ae6b13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.902857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.902760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acacd1221e4ccabdaec949b2a26be37f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"acacd1221e4ccabdaec949b2a26be37f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.902857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.902782 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acacd1221e4ccabdaec949b2a26be37f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"acacd1221e4ccabdaec949b2a26be37f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.902857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.902833 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acacd1221e4ccabdaec949b2a26be37f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"acacd1221e4ccabdaec949b2a26be37f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.902857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.902843 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/acacd1221e4ccabdaec949b2a26be37f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal\" (UID: \"acacd1221e4ccabdaec949b2a26be37f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.902976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:04.902833 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/420479272364a70b28d1192701ae6b13-config\") pod \"kube-apiserver-proxy-ip-10-0-131-134.ec2.internal\" (UID: \"420479272364a70b28d1192701ae6b13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Jul 21 08:19:04.978207 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:04.978133 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.078618 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.078585 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.113170 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.113131 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:05.117427 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.117236 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Jul 21 08:19:05.179093 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.179054 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.282607 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.282528 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.383606 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.383565 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.437079 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.437054 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 08:19:05.437670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.437194 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 08:19:05.437670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.437244 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 08:19:05.484064 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.484034 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.498522 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.498488 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 08:14:05 +0000 UTC" deadline="2028-01-22 16:28:21.562785146 +0000 UTC" Jul 21 08:19:05.498522 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.498517 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13208h9m16.064271296s" Jul 21 08:19:05.584864 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.584835 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.684946 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.684919 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.685092 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:05.684927 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacacd1221e4ccabdaec949b2a26be37f.slice/crio-64d58c7a345d911c8fded5064f9673966465cebb66f96783e157837f32341687 WatchSource:0}: Error finding container 64d58c7a345d911c8fded5064f9673966465cebb66f96783e157837f32341687: Status 404 returned error can't find the container with id 64d58c7a345d911c8fded5064f9673966465cebb66f96783e157837f32341687 Jul 21 08:19:05.685335 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:05.685318 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420479272364a70b28d1192701ae6b13.slice/crio-9775cabff0bba7e418a30f0316de474655023e9bb6ee875c9f14dbfc1c5ff0e4 WatchSource:0}: Error finding container 9775cabff0bba7e418a30f0316de474655023e9bb6ee875c9f14dbfc1c5ff0e4: Status 404 returned error can't find the container with id 9775cabff0bba7e418a30f0316de474655023e9bb6ee875c9f14dbfc1c5ff0e4 Jul 21 08:19:05.690133 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.690114 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:19:05.791393 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:05.791349 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-134.ec2.internal\" not found" Jul 21 08:19:05.848375 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.848270 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:19:05.873045 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.873016 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:19:05.900083 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.900049 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" Jul 21 08:19:05.910524 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.910498 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 08:19:05.911684 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.911668 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" Jul 21 08:19:05.919771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:05.919753 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 08:19:06.476902 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.476871 2553 apiserver.go:52] "Watching apiserver" Jul 21 08:19:06.483330 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.483287 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 08:19:06.485432 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.485389 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-9m59z","openshift-network-operator/iptables-alerter-f7rrk","openshift-ovn-kubernetes/ovnkube-node-bf86t","kube-system/global-pull-secret-syncer-2ffz9","kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lsd2c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal","openshift-multus/multus-additional-cni-plugins-mjksz","openshift-network-diagnostics/network-check-target-72jw7","kube-system/konnectivity-agent-pfwh9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9","openshift-dns/node-resolver-ksjr8","openshift-image-registry/node-ca-bxvjt","openshift-multus/multus-tbtpj"] Jul 21 08:19:06.487439 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.487417 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.489667 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.489350 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.489667 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.489658 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-52k75\"" Jul 21 08:19:06.489867 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.489803 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 08:19:06.489867 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.489855 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 08:19:06.491509 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.491487 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 08:19:06.492440 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.491909 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 08:19:06.492440 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.491968 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.492440 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.492015 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jb2fk\"" Jul 21 08:19:06.492440 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.491925 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 08:19:06.493058 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.493039 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.493144 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.493120 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:06.494354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.494335 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 08:19:06.494976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.494589 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 08:19:06.494976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.494629 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 08:19:06.494976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.494651 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.494976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.494723 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f4fp4\"" Jul 21 08:19:06.494976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.494858 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 08:19:06.495228 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.494984 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 08:19:06.495228 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.495162 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 08:19:06.496344 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.496325 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.496670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.496646 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 08:19:06.497630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.497424 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 08:19:06.497630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.497555 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 08:19:06.497630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.497575 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 08:19:06.497808 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.497659 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b6gpp\"" Jul 21 08:19:06.497986 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.497967 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:06.498070 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.498037 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:06.498477 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.498459 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:19:06.498580 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.498501 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 08:19:06.498580 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.498548 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pncr8\"" Jul 21 08:19:06.499601 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.499581 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.500800 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.499731 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 08:19:06.501551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.501531 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 08:19:06.501649 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.501555 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.501910 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.501893 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 08:19:06.501910 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.501907 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 08:19:06.502169 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.502141 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhbf2\"" Jul 21 08:19:06.503096 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.503077 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:06.503187 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.503145 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:06.505131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.503908 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 08:19:06.505131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.504587 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.505751 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.505311 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 08:19:06.505751 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.505305 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vztbx\"" Jul 21 08:19:06.505751 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.505346 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 08:19:06.506435 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.506419 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.507174 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.507022 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 08:19:06.507174 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.507052 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h9khx\"" Jul 21 08:19:06.507174 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.507086 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 08:19:06.508684 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.508658 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 08:19:06.508784 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.508664 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 08:19:06.508784 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.508717 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x5ftn\"" Jul 21 08:19:06.513048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.513026 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.513940 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-kubelet\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.513985 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d000c11a-72c8-4445-9dc3-36403935205c-cni-binary-copy\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514021 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysconfig\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514053 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-lib-modules\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514080 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-host\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514111 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-device-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514151 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-cni-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514181 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-os-release\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514212 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-tuned\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514246 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4db92d1-f92d-4d7d-a8d9-90a99829883b-tmp\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514282 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514332 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-slash\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514365 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d000c11a-72c8-4445-9dc3-36403935205c-multus-daemon-config\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514400 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-systemd\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514428 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514461 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhz7\" (UniqueName: \"kubernetes.io/projected/b20b6770-0988-4982-9540-740ef9289986-kube-api-access-nnhz7\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.515949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514491 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-k8s-cni-cncf-io\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514524 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-systemd\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514557 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-run-ovn-kubernetes\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514585 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-cnibin\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514613 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-socket-dir-parent\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514650 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514678 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-node-log\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514708 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8a24c11-99e9-4221-b50b-851f24298d13-kubelet-config\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514737 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-netns\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514763 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-etc-kubernetes\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514794 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-modprobe-d\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514830 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2090af5-b4b4-457b-8a32-8e361407a9ea-serviceca\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514859 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-socket-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514883 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-cni-bin\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514912 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2090af5-b4b4-457b-8a32-8e361407a9ea-host\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514941 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7dj\" (UniqueName: \"kubernetes.io/projected/f2090af5-b4b4-457b-8a32-8e361407a9ea-kube-api-access-ds7dj\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.514972 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-ovnkube-config\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.516710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515005 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-registration-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515041 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-ovn\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515072 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-kubelet\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515099 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515130 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-system-cni-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515160 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-kubernetes\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515191 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-log-socket\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515221 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-cni-netd\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515246 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-ovnkube-script-lib\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515278 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-multus-certs\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515307 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d35d5d7a-78b8-4f4c-947d-d738b21f2034-agent-certs\") pod \"konnectivity-agent-pfwh9\" (UID: \"d35d5d7a-78b8-4f4c-947d-d738b21f2034\") " pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515336 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-hostroot\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515361 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqlwl\" (UniqueName: \"kubernetes.io/projected/d000c11a-72c8-4445-9dc3-36403935205c-kube-api-access-hqlwl\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515410 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysctl-conf\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515442 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-run\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515479 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-etc-selinux\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515510 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-sys-fs\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.517354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515541 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8a24c11-99e9-4221-b50b-851f24298d13-dbus\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515576 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d35d5d7a-78b8-4f4c-947d-d738b21f2034-konnectivity-ca\") pod \"konnectivity-agent-pfwh9\" (UID: \"d35d5d7a-78b8-4f4c-947d-d738b21f2034\") " pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515608 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f87f794-2bb1-4ada-9342-48a583e8f7a9-iptables-alerter-script\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515639 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f87f794-2bb1-4ada-9342-48a583e8f7a9-host-slash\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515668 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-systemd-units\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515692 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-conf-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515724 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-run-netns\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515756 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-var-lib-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515791 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-env-overrides\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515818 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b20b6770-0988-4982-9540-740ef9289986-ovn-node-metrics-cert\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515854 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-cni-bin\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.515912 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-cni-multus\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.516280 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysctl-d\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.516845 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlxzk\" (UniqueName: \"kubernetes.io/projected/3f87f794-2bb1-4ada-9342-48a583e8f7a9-kube-api-access-qlxzk\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.516909 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vtkj\" (UniqueName: \"kubernetes.io/projected/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-kube-api-access-8vtkj\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.516975 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-etc-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.518006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.517022 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-sys\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.518603 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.517059 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-var-lib-kubelet\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.518603 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.517095 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czbp\" (UniqueName: \"kubernetes.io/projected/f4db92d1-f92d-4d7d-a8d9-90a99829883b-kube-api-access-8czbp\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.518603 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.517130 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:06.534569 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.534544 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dc464" Jul 21 08:19:06.541802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.541780 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dc464" Jul 21 08:19:06.601004 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.600972 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 08:19:06.617265 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617234 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:19:06.617461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617369 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-run\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.617461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617426 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-etc-selinux\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.617461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617453 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-run\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617457 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-sys-fs\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617498 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8a24c11-99e9-4221-b50b-851f24298d13-dbus\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617517 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d35d5d7a-78b8-4f4c-947d-d738b21f2034-konnectivity-ca\") pod \"konnectivity-agent-pfwh9\" (UID: \"d35d5d7a-78b8-4f4c-947d-d738b21f2034\") " pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617520 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-sys-fs\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617538 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f87f794-2bb1-4ada-9342-48a583e8f7a9-iptables-alerter-script\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617556 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f87f794-2bb1-4ada-9342-48a583e8f7a9-host-slash\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617582 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxc26\" (UniqueName: \"kubernetes.io/projected/d775c285-f2a3-42b5-8867-88a0faefff5f-kube-api-access-xxc26\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617601 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-cnibin\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617618 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-systemd-units\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617635 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-conf-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.617630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617633 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f87f794-2bb1-4ada-9342-48a583e8f7a9-host-slash\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617597 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-etc-selinux\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617717 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-systemd-units\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617746 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8a24c11-99e9-4221-b50b-851f24298d13-dbus\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617750 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-conf-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617765 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nns4m\" (UniqueName: \"kubernetes.io/projected/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-kube-api-access-nns4m\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617799 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmjf\" (UniqueName: \"kubernetes.io/projected/703d8cfe-167c-4860-b63d-517f2f57267e-kube-api-access-9nmjf\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617828 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-run-netns\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-var-lib-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617881 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-env-overrides\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617903 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b20b6770-0988-4982-9540-740ef9289986-ovn-node-metrics-cert\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617927 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-cni-bin\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617966 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-cni-multus\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.617992 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysctl-d\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618017 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlxzk\" (UniqueName: \"kubernetes.io/projected/3f87f794-2bb1-4ada-9342-48a583e8f7a9-kube-api-access-qlxzk\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618045 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vtkj\" (UniqueName: \"kubernetes.io/projected/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-kube-api-access-8vtkj\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618077 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.618178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618105 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-etc-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618132 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-sys\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618157 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d35d5d7a-78b8-4f4c-947d-d738b21f2034-konnectivity-ca\") pod \"konnectivity-agent-pfwh9\" (UID: \"d35d5d7a-78b8-4f4c-947d-d738b21f2034\") " pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618200 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-cni-multus\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618157 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f87f794-2bb1-4ada-9342-48a583e8f7a9-iptables-alerter-script\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618221 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-var-lib-kubelet\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618161 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-var-lib-kubelet\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618256 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-run-netns\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618261 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czbp\" (UniqueName: \"kubernetes.io/projected/f4db92d1-f92d-4d7d-a8d9-90a99829883b-kube-api-access-8czbp\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618301 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-var-lib-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618329 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618346 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysctl-d\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618354 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-kubelet\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618527 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d000c11a-72c8-4445-9dc3-36403935205c-cni-binary-copy\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618542 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-sys\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618562 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysconfig\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-kubelet\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618592 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-lib-modules\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618618 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-host\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618642 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-device-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618670 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-cni-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618694 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-os-release\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618717 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-tuned\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4db92d1-f92d-4d7d-a8d9-90a99829883b-tmp\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618758 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-host\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618766 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d775c285-f2a3-42b5-8867-88a0faefff5f-hosts-file\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618809 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618832 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-device-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618867 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618897 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-cni-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618896 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-slash\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618935 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d000c11a-72c8-4445-9dc3-36403935205c-multus-daemon-config\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618938 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-slash\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-os-release\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618988 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.619995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.618995 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysconfig\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619014 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619041 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-cni-bin\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619044 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-systemd\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619098 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-systemd\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619104 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-env-overrides\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619162 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-os-release\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619244 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d000c11a-72c8-4445-9dc3-36403935205c-cni-binary-copy\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619296 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619327 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhz7\" (UniqueName: \"kubernetes.io/projected/b20b6770-0988-4982-9540-740ef9289986-kube-api-access-nnhz7\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619406 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619448 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-k8s-cni-cncf-io\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619497 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-systemd\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619499 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619537 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619576 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-run-ovn-kubernetes\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619605 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-cnibin\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619606 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-k8s-cni-cncf-io\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.620904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619631 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-systemd\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619658 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-socket-dir-parent\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619674 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-cnibin\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-etc-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619693 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d775c285-f2a3-42b5-8867-88a0faefff5f-tmp-dir\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619717 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-run-ovn-kubernetes\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619790 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619817 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-node-log\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619838 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-lib-modules\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619844 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8a24c11-99e9-4221-b50b-851f24298d13-kubelet-config\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619884 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-netns\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619892 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d000c11a-72c8-4445-9dc3-36403935205c-multus-daemon-config\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619905 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-etc-kubernetes\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619914 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-multus-socket-dir-parent\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619932 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-modprobe-d\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.619956 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-openvswitch\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620050 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2090af5-b4b4-457b-8a32-8e361407a9ea-serviceca\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620062 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-modprobe-d\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.621980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620078 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-socket-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620079 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-netns\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620102 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-cni-bin\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620104 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-etc-kubernetes\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2090af5-b4b4-457b-8a32-8e361407a9ea-host\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-node-log\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620149 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-cni-bin\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620161 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7dj\" (UniqueName: \"kubernetes.io/projected/f2090af5-b4b4-457b-8a32-8e361407a9ea-kube-api-access-ds7dj\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620199 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2090af5-b4b4-457b-8a32-8e361407a9ea-host\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620214 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8a24c11-99e9-4221-b50b-851f24298d13-kubelet-config\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620258 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-ovnkube-config\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620301 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-socket-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-registration-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620371 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-ovn\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620412 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-kubelet\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620473 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-run-ovn\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620528 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620558 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-system-cni-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.623038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620600 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2090af5-b4b4-457b-8a32-8e361407a9ea-serviceca\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620620 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-kubernetes\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620654 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-var-lib-kubelet\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620658 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620698 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-system-cni-dir\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620706 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-registration-dir\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620738 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-system-cni-dir\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-log-socket\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.620791 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620801 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-cni-netd\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620834 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-ovnkube-script-lib\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620874 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-host-cni-netd\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.620883 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret podName:d8a24c11-99e9-4221-b50b-851f24298d13 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:07.120863071 +0000 UTC m=+3.006533630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret") pod "global-pull-secret-syncer-2ffz9" (UID: "d8a24c11-99e9-4221-b50b-851f24298d13") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620866 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-multus-certs\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620805 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-kubernetes\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620921 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d35d5d7a-78b8-4f4c-947d-d738b21f2034-agent-certs\") pod \"konnectivity-agent-pfwh9\" (UID: \"d35d5d7a-78b8-4f4c-947d-d738b21f2034\") " pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620923 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b20b6770-0988-4982-9540-740ef9289986-log-socket\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.623954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.620947 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-host-run-multus-certs\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.621119 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-hostroot\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.621173 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqlwl\" (UniqueName: \"kubernetes.io/projected/d000c11a-72c8-4445-9dc3-36403935205c-kube-api-access-hqlwl\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.621203 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysctl-conf\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.621309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-ovnkube-config\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.621428 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-sysctl-conf\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.621480 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d000c11a-72c8-4445-9dc3-36403935205c-hostroot\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.621514 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b20b6770-0988-4982-9540-740ef9289986-ovnkube-script-lib\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.623636 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b20b6770-0988-4982-9540-740ef9289986-ovn-node-metrics-cert\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.623867 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d35d5d7a-78b8-4f4c-947d-d738b21f2034-agent-certs\") pod \"konnectivity-agent-pfwh9\" (UID: \"d35d5d7a-78b8-4f4c-947d-d738b21f2034\") " pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.624087 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4db92d1-f92d-4d7d-a8d9-90a99829883b-tmp\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.624641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.624105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4db92d1-f92d-4d7d-a8d9-90a99829883b-etc-tuned\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.627472 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.627373 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:06.627636 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.627619 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:06.627825 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.627778 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6p27d for pod openshift-network-diagnostics/network-check-target-72jw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:06.627940 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.627924 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d podName:b8ebe912-4210-47a1-b6db-e301c388eb82 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:07.127902977 +0000 UTC m=+3.013573522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6p27d" (UniqueName: "kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d") pod "network-check-target-72jw7" (UID: "b8ebe912-4210-47a1-b6db-e301c388eb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:06.630445 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.630421 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlxzk\" (UniqueName: \"kubernetes.io/projected/3f87f794-2bb1-4ada-9342-48a583e8f7a9-kube-api-access-qlxzk\") pod \"iptables-alerter-f7rrk\" (UID: \"3f87f794-2bb1-4ada-9342-48a583e8f7a9\") " pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.630560 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.630474 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vtkj\" (UniqueName: \"kubernetes.io/projected/f835cf8c-f64a-49ca-bef2-2e2a5f84f458-kube-api-access-8vtkj\") pod \"aws-ebs-csi-driver-node-lfzq9\" (UID: \"f835cf8c-f64a-49ca-bef2-2e2a5f84f458\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.630750 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.630711 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czbp\" (UniqueName: \"kubernetes.io/projected/f4db92d1-f92d-4d7d-a8d9-90a99829883b-kube-api-access-8czbp\") pod \"tuned-lsd2c\" (UID: \"f4db92d1-f92d-4d7d-a8d9-90a99829883b\") " pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.631892 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.631862 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7dj\" (UniqueName: \"kubernetes.io/projected/f2090af5-b4b4-457b-8a32-8e361407a9ea-kube-api-access-ds7dj\") pod \"node-ca-bxvjt\" (UID: \"f2090af5-b4b4-457b-8a32-8e361407a9ea\") " pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.632498 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.632478 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqlwl\" (UniqueName: \"kubernetes.io/projected/d000c11a-72c8-4445-9dc3-36403935205c-kube-api-access-hqlwl\") pod \"multus-tbtpj\" (UID: \"d000c11a-72c8-4445-9dc3-36403935205c\") " pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.632666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.632649 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhz7\" (UniqueName: \"kubernetes.io/projected/b20b6770-0988-4982-9540-740ef9289986-kube-api-access-nnhz7\") pod \"ovnkube-node-bf86t\" (UID: \"b20b6770-0988-4982-9540-740ef9289986\") " pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.673088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.673033 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" event={"ID":"acacd1221e4ccabdaec949b2a26be37f","Type":"ContainerStarted","Data":"64d58c7a345d911c8fded5064f9673966465cebb66f96783e157837f32341687"} Jul 21 08:19:06.674401 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.674366 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" event={"ID":"420479272364a70b28d1192701ae6b13","Type":"ContainerStarted","Data":"9775cabff0bba7e418a30f0316de474655023e9bb6ee875c9f14dbfc1c5ff0e4"} Jul 21 08:19:06.722534 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722497 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.722748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722558 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d775c285-f2a3-42b5-8867-88a0faefff5f-hosts-file\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.722748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722586 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.722748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722615 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-os-release\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.722748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722634 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.722748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722652 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.722748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722672 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d775c285-f2a3-42b5-8867-88a0faefff5f-hosts-file\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.723127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.722682 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d775c285-f2a3-42b5-8867-88a0faefff5f-tmp-dir\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.723127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723051 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:06.723127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723084 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-system-cni-dir\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723126 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxc26\" (UniqueName: \"kubernetes.io/projected/d775c285-f2a3-42b5-8867-88a0faefff5f-kube-api-access-xxc26\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.723341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723152 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-cnibin\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723168 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nns4m\" (UniqueName: \"kubernetes.io/projected/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-kube-api-access-nns4m\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:06.723341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723211 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmjf\" (UniqueName: \"kubernetes.io/projected/703d8cfe-167c-4860-b63d-517f2f57267e-kube-api-access-9nmjf\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723246 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723260 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-os-release\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723283 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723727 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723369 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-cnibin\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723727 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.723512 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:06.723727 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723547 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d775c285-f2a3-42b5-8867-88a0faefff5f-tmp-dir\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.723727 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:06.723596 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:19:07.22357617 +0000 UTC m=+3.109246699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:06.723727 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723608 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/703d8cfe-167c-4860-b63d-517f2f57267e-system-cni-dir\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.723991 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.723805 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/703d8cfe-167c-4860-b63d-517f2f57267e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.732084 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.732011 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nns4m\" (UniqueName: \"kubernetes.io/projected/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-kube-api-access-nns4m\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:06.732319 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.732175 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxc26\" (UniqueName: \"kubernetes.io/projected/d775c285-f2a3-42b5-8867-88a0faefff5f-kube-api-access-xxc26\") pod \"node-resolver-ksjr8\" (UID: \"d775c285-f2a3-42b5-8867-88a0faefff5f\") " pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.732916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.732893 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmjf\" (UniqueName: \"kubernetes.io/projected/703d8cfe-167c-4860-b63d-517f2f57267e-kube-api-access-9nmjf\") pod \"multus-additional-cni-plugins-mjksz\" (UID: \"703d8cfe-167c-4860-b63d-517f2f57267e\") " pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.801835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.801795 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:06.808841 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.808814 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" Jul 21 08:19:06.820170 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.819948 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:06.824371 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.823874 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbtpj" Jul 21 08:19:06.830242 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.830037 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" Jul 21 08:19:06.835994 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.835972 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bxvjt" Jul 21 08:19:06.841795 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.841770 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f7rrk" Jul 21 08:19:06.850104 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.850081 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mjksz" Jul 21 08:19:06.855418 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.855394 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ksjr8" Jul 21 08:19:06.953035 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:06.952802 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 08:19:07.100830 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.100765 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703d8cfe_167c_4860_b63d_517f2f57267e.slice/crio-e0a7ecbe02bd85d82bbf804b2f67c7c3d689d6feb2f04a567fadcaa790ef8c54 WatchSource:0}: Error finding container e0a7ecbe02bd85d82bbf804b2f67c7c3d689d6feb2f04a567fadcaa790ef8c54: Status 404 returned error can't find the container with id e0a7ecbe02bd85d82bbf804b2f67c7c3d689d6feb2f04a567fadcaa790ef8c54 Jul 21 08:19:07.101703 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.101678 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd775c285_f2a3_42b5_8867_88a0faefff5f.slice/crio-7063eb57df5086399fb4ec41eca691e9695ec90234c94103d1764ad5df413d98 WatchSource:0}: Error finding container 7063eb57df5086399fb4ec41eca691e9695ec90234c94103d1764ad5df413d98: Status 404 returned error can't find the container with id 7063eb57df5086399fb4ec41eca691e9695ec90234c94103d1764ad5df413d98 Jul 21 08:19:07.109281 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.109253 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35d5d7a_78b8_4f4c_947d_d738b21f2034.slice/crio-8978d305acff1354115f504b84e739eefa2b4266b3dfc4fd13edd63cd697bdf0 WatchSource:0}: Error finding container 8978d305acff1354115f504b84e739eefa2b4266b3dfc4fd13edd63cd697bdf0: Status 404 returned error can't find the container with id 8978d305acff1354115f504b84e739eefa2b4266b3dfc4fd13edd63cd697bdf0 Jul 21 08:19:07.110096 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.110065 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd000c11a_72c8_4445_9dc3_36403935205c.slice/crio-8a5473d5a3a35832fb071db24cb35e60ef7be28d63ae10cab8210f56e12f4e79 WatchSource:0}: Error finding container 8a5473d5a3a35832fb071db24cb35e60ef7be28d63ae10cab8210f56e12f4e79: Status 404 returned error can't find the container with id 8a5473d5a3a35832fb071db24cb35e60ef7be28d63ae10cab8210f56e12f4e79 Jul 21 08:19:07.111227 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.111199 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4db92d1_f92d_4d7d_a8d9_90a99829883b.slice/crio-b386b4aeb3f4622070a33f1de7f97716c2fdf5152cb3318e16911c99d549a9dd WatchSource:0}: Error finding container b386b4aeb3f4622070a33f1de7f97716c2fdf5152cb3318e16911c99d549a9dd: Status 404 returned error can't find the container with id b386b4aeb3f4622070a33f1de7f97716c2fdf5152cb3318e16911c99d549a9dd Jul 21 08:19:07.112531 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.112503 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f87f794_2bb1_4ada_9342_48a583e8f7a9.slice/crio-77d0d4ac3f92141863519d142ccef3504123590849a861aa69cd0fc6d2ff80a8 WatchSource:0}: Error finding container 77d0d4ac3f92141863519d142ccef3504123590849a861aa69cd0fc6d2ff80a8: Status 404 returned error can't find the container with id 77d0d4ac3f92141863519d142ccef3504123590849a861aa69cd0fc6d2ff80a8 Jul 21 08:19:07.113361 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.113332 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf835cf8c_f64a_49ca_bef2_2e2a5f84f458.slice/crio-108888d0972b9264efc7a063d73fde145b07201746b8ba996782bf73f2ee40f3 WatchSource:0}: Error finding container 108888d0972b9264efc7a063d73fde145b07201746b8ba996782bf73f2ee40f3: Status 404 returned error can't find the container with id 108888d0972b9264efc7a063d73fde145b07201746b8ba996782bf73f2ee40f3 Jul 21 08:19:07.114714 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.113886 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20b6770_0988_4982_9540_740ef9289986.slice/crio-0a751ede05e69320580d761638517456f667411b2234c35d9f768fdcf76b1e06 WatchSource:0}: Error finding container 0a751ede05e69320580d761638517456f667411b2234c35d9f768fdcf76b1e06: Status 404 returned error can't find the container with id 0a751ede05e69320580d761638517456f667411b2234c35d9f768fdcf76b1e06 Jul 21 08:19:07.115528 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:07.115323 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2090af5_b4b4_457b_8a32_8e361407a9ea.slice/crio-e5e57b2768056015cec66194a8e06a9262fc4f516e6f1ba6611b5467ef6f64c2 WatchSource:0}: Error finding container e5e57b2768056015cec66194a8e06a9262fc4f516e6f1ba6611b5467ef6f64c2: Status 404 returned error can't find the container with id e5e57b2768056015cec66194a8e06a9262fc4f516e6f1ba6611b5467ef6f64c2 Jul 21 08:19:07.127041 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.127015 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:07.127183 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.127166 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:07.127237 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.127226 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret podName:d8a24c11-99e9-4221-b50b-851f24298d13 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:08.127209568 +0000 UTC m=+4.012880097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret") pod "global-pull-secret-syncer-2ffz9" (UID: "d8a24c11-99e9-4221-b50b-851f24298d13") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:07.228170 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.228136 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:07.228340 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.228214 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:07.228340 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.228304 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:07.228549 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.228380 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:07.228549 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.228401 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:07.228549 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.228403 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:19:08.228380201 +0000 UTC m=+4.114050748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:07.228549 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.228411 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6p27d for pod openshift-network-diagnostics/network-check-target-72jw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:07.228549 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.228457 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d podName:b8ebe912-4210-47a1-b6db-e301c388eb82 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:08.228440505 +0000 UTC m=+4.114111034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6p27d" (UniqueName: "kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d") pod "network-check-target-72jw7" (UID: "b8ebe912-4210-47a1-b6db-e301c388eb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:07.545215 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.544968 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 08:14:07 +0000 UTC" deadline="2028-03-02 17:18:02.858787416 +0000 UTC" Jul 21 08:19:07.545215 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.545213 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14168h58m55.313582535s" Jul 21 08:19:07.668441 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.668403 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:07.668640 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.668404 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:07.668640 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.668555 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:07.668795 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:07.668638 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:07.678464 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.678432 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbtpj" event={"ID":"d000c11a-72c8-4445-9dc3-36403935205c","Type":"ContainerStarted","Data":"8a5473d5a3a35832fb071db24cb35e60ef7be28d63ae10cab8210f56e12f4e79"} Jul 21 08:19:07.679932 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.679900 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pfwh9" event={"ID":"d35d5d7a-78b8-4f4c-947d-d738b21f2034","Type":"ContainerStarted","Data":"8978d305acff1354115f504b84e739eefa2b4266b3dfc4fd13edd63cd697bdf0"} Jul 21 08:19:07.683857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.683800 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f7rrk" event={"ID":"3f87f794-2bb1-4ada-9342-48a583e8f7a9","Type":"ContainerStarted","Data":"77d0d4ac3f92141863519d142ccef3504123590849a861aa69cd0fc6d2ff80a8"} Jul 21 08:19:07.685163 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.685128 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksjr8" event={"ID":"d775c285-f2a3-42b5-8867-88a0faefff5f","Type":"ContainerStarted","Data":"7063eb57df5086399fb4ec41eca691e9695ec90234c94103d1764ad5df413d98"} Jul 21 08:19:07.687883 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.687829 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerStarted","Data":"e0a7ecbe02bd85d82bbf804b2f67c7c3d689d6feb2f04a567fadcaa790ef8c54"} Jul 21 08:19:07.691022 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.690332 2553 generic.go:356] "Generic (PLEG): container finished" podID="acacd1221e4ccabdaec949b2a26be37f" containerID="af8528efba5b36697045471a49ea5351828815e02485bf062132e570caaa0eec" exitCode=0 Jul 21 08:19:07.691022 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.690430 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" event={"ID":"acacd1221e4ccabdaec949b2a26be37f","Type":"ContainerDied","Data":"af8528efba5b36697045471a49ea5351828815e02485bf062132e570caaa0eec"} Jul 21 08:19:07.692943 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.692913 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bxvjt" event={"ID":"f2090af5-b4b4-457b-8a32-8e361407a9ea","Type":"ContainerStarted","Data":"e5e57b2768056015cec66194a8e06a9262fc4f516e6f1ba6611b5467ef6f64c2"} Jul 21 08:19:07.694599 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.694518 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" event={"ID":"f835cf8c-f64a-49ca-bef2-2e2a5f84f458","Type":"ContainerStarted","Data":"108888d0972b9264efc7a063d73fde145b07201746b8ba996782bf73f2ee40f3"} Jul 21 08:19:07.696863 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.696838 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"0a751ede05e69320580d761638517456f667411b2234c35d9f768fdcf76b1e06"} Jul 21 08:19:07.698628 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:07.698606 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" event={"ID":"f4db92d1-f92d-4d7d-a8d9-90a99829883b","Type":"ContainerStarted","Data":"b386b4aeb3f4622070a33f1de7f97716c2fdf5152cb3318e16911c99d549a9dd"} Jul 21 08:19:08.138374 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.137889 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:08.138374 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.138035 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:08.138374 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.138094 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret podName:d8a24c11-99e9-4221-b50b-851f24298d13 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:10.138074663 +0000 UTC m=+6.023745198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret") pod "global-pull-secret-syncer-2ffz9" (UID: "d8a24c11-99e9-4221-b50b-851f24298d13") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.238800 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.238886 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.239029 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.239095 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:19:10.239073985 +0000 UTC m=+6.124744514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.239195 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.239227 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.239240 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6p27d for pod openshift-network-diagnostics/network-check-target-72jw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:08.239748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.239280 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d podName:b8ebe912-4210-47a1-b6db-e301c388eb82 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:10.239267479 +0000 UTC m=+6.124938014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6p27d" (UniqueName: "kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d") pod "network-check-target-72jw7" (UID: "b8ebe912-4210-47a1-b6db-e301c388eb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:08.545588 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.545464 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 08:14:07 +0000 UTC" deadline="2028-02-12 05:51:03.515704629 +0000 UTC" Jul 21 08:19:08.545588 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.545535 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13701h31m54.970174316s" Jul 21 08:19:08.668904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.668348 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:08.668904 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:08.668517 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:08.712949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.712211 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" event={"ID":"acacd1221e4ccabdaec949b2a26be37f","Type":"ContainerStarted","Data":"eb3eb49169022e3ded16b72500f51f1daf57a18322ed12db6d52d2e6ab9a89c9"} Jul 21 08:19:08.724998 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.724516 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" event={"ID":"420479272364a70b28d1192701ae6b13","Type":"ContainerStarted","Data":"8960ba24808f329f094923e9a30b2a4e300c981fcf4e4fdfb2c1e53430c8cf33"} Jul 21 08:19:08.731895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:08.731837 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-134.ec2.internal" podStartSLOduration=2.731821776 podStartE2EDuration="2.731821776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:08.73099944 +0000 UTC m=+4.616669993" watchObservedRunningTime="2026-07-21 08:19:08.731821776 +0000 UTC m=+4.617492330" Jul 21 08:19:09.668045 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:09.668004 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:09.668568 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:09.668159 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:09.668657 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:09.668639 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:09.669205 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:09.668750 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:10.156991 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:10.156815 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:10.156991 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.156982 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:10.157214 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.157050 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret podName:d8a24c11-99e9-4221-b50b-851f24298d13 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:14.157030602 +0000 UTC m=+10.042701158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret") pod "global-pull-secret-syncer-2ffz9" (UID: "d8a24c11-99e9-4221-b50b-851f24298d13") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:10.258097 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:10.258193 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.258366 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.258442 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:19:14.258420099 +0000 UTC m=+10.144090652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.258978 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.259003 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.259017 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6p27d for pod openshift-network-diagnostics/network-check-target-72jw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:10.259115 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.259072 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d podName:b8ebe912-4210-47a1-b6db-e301c388eb82 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:14.259054321 +0000 UTC m=+10.144724874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6p27d" (UniqueName: "kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d") pod "network-check-target-72jw7" (UID: "b8ebe912-4210-47a1-b6db-e301c388eb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:10.670557 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:10.670522 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:10.671055 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:10.670694 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:11.668530 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:11.668500 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:11.668750 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:11.668609 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:11.668750 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:11.668657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:11.668750 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:11.668700 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:12.668580 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:12.667957 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:12.668580 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:12.668119 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:13.668677 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:13.668644 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:13.669210 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:13.668644 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:13.669210 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:13.668780 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:13.669210 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:13.668904 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:14.193891 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:14.193775 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:14.194093 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.193969 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:14.194093 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.194068 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret podName:d8a24c11-99e9-4221-b50b-851f24298d13 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:22.194043703 +0000 UTC m=+18.079714251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret") pod "global-pull-secret-syncer-2ffz9" (UID: "d8a24c11-99e9-4221-b50b-851f24298d13") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:14.295046 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:14.295007 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:14.295236 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:14.295100 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:14.295236 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.295226 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:14.295381 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.295294 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:19:22.295272059 +0000 UTC m=+18.180942590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:14.295820 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.295763 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:14.295820 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.295793 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:14.295820 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.295806 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6p27d for pod openshift-network-diagnostics/network-check-target-72jw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:14.296056 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.295853 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d podName:b8ebe912-4210-47a1-b6db-e301c388eb82 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:22.295837717 +0000 UTC m=+18.181508247 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6p27d" (UniqueName: "kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d") pod "network-check-target-72jw7" (UID: "b8ebe912-4210-47a1-b6db-e301c388eb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:14.671840 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:14.671765 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:14.672286 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:14.671888 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:15.668996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:15.668503 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:15.668996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:15.668555 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:15.668996 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:15.668667 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:15.668996 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:15.668762 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:16.670217 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:16.670170 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:16.670694 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:16.670333 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:17.668383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:17.668035 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:17.668383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:17.668035 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:17.668383 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:17.668189 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:17.668383 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:17.668258 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:18.667881 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:18.667800 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:18.668296 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:18.667933 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:19.668263 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:19.668233 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:19.668263 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:19.668255 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:19.668771 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:19.668368 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:19.668771 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:19.668419 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:20.670443 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:20.670408 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:20.670875 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:20.670542 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:21.668278 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:21.668242 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:21.668478 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:21.668288 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:21.668478 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:21.668377 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:21.668580 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:21.668495 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:22.254702 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:22.254611 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:22.255095 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.254790 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:22.255095 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.254882 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret podName:d8a24c11-99e9-4221-b50b-851f24298d13 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:38.254858566 +0000 UTC m=+34.140529114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret") pod "global-pull-secret-syncer-2ffz9" (UID: "d8a24c11-99e9-4221-b50b-851f24298d13") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:22.355194 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:22.355162 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:22.355413 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:22.355232 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:22.355413 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.355366 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:22.355413 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.355381 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:22.355413 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.355404 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:22.355413 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.355417 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6p27d for pod openshift-network-diagnostics/network-check-target-72jw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:22.355701 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.355444 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:19:38.355423092 +0000 UTC m=+34.241093618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:22.355701 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.355468 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d podName:b8ebe912-4210-47a1-b6db-e301c388eb82 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:38.355452184 +0000 UTC m=+34.241122723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6p27d" (UniqueName: "kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d") pod "network-check-target-72jw7" (UID: "b8ebe912-4210-47a1-b6db-e301c388eb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:22.668598 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:22.668510 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:22.668763 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:22.668646 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:23.668326 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:23.668281 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:23.668710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:23.668323 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:23.668710 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:23.668432 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:23.668710 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:23.668550 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:24.669180 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:24.669147 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:24.669615 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:24.669287 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:25.668423 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:25.668389 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:25.668590 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:25.668400 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:25.668590 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:25.668512 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:25.668678 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:25.668590 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:26.670201 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.668329 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:26.670201 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:26.668646 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:26.772559 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.772527 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbtpj" event={"ID":"d000c11a-72c8-4445-9dc3-36403935205c","Type":"ContainerStarted","Data":"44ae886e227268c26c72e47157cf0a3a4bbe2ab6ef42c760efe3f43cc72042e9"} Jul 21 08:19:26.773837 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.773813 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pfwh9" event={"ID":"d35d5d7a-78b8-4f4c-947d-d738b21f2034","Type":"ContainerStarted","Data":"da5042ae670fe78ef2fb5d1461bb4383ec761d15372b8ef1b4490b505eb113ee"} Jul 21 08:19:26.774824 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.774798 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ksjr8" event={"ID":"d775c285-f2a3-42b5-8867-88a0faefff5f","Type":"ContainerStarted","Data":"fd32a19fdd8dd73748dda1b23e2dcd0a077c319fbd1c0b976cfeadd09584dd7e"} Jul 21 08:19:26.776194 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.776146 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerStarted","Data":"6b9d043b8eac07308de6fd60bd6ad24d656a6142c9c26fb829e1ca0e648690da"} Jul 21 08:19:26.777501 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.777461 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bxvjt" event={"ID":"f2090af5-b4b4-457b-8a32-8e361407a9ea","Type":"ContainerStarted","Data":"d116aa7ea424465a609c509c7eb6c51245a9bac520e901e21fa2c526401313ac"} Jul 21 08:19:26.779239 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.779204 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" event={"ID":"f835cf8c-f64a-49ca-bef2-2e2a5f84f458","Type":"ContainerStarted","Data":"f514ebc2581990ede944de0eed930d2fe9569980355006c0d71d12d06b6f658a"} Jul 21 08:19:26.785532 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.785505 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"4f2f02875aa78cff76b8e9fa40978f05809a7150ef806d880fcf2fee4d1dde35"} Jul 21 08:19:26.786895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.786875 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" event={"ID":"f4db92d1-f92d-4d7d-a8d9-90a99829883b","Type":"ContainerStarted","Data":"e6a9c09550f7d6508d0018804c36687fc92d02c01d50bec390138f2f3084cd2b"} Jul 21 08:19:26.797132 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.797085 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-134.ec2.internal" podStartSLOduration=20.797074442 podStartE2EDuration="20.797074442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:08.744653291 +0000 UTC m=+4.630323843" watchObservedRunningTime="2026-07-21 08:19:26.797074442 +0000 UTC m=+22.682744993" Jul 21 08:19:26.810438 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.810389 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lsd2c" podStartSLOduration=2.397297677 podStartE2EDuration="21.810372133s" totalImagesPullingTime="19.413074456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:26.809761077 +0000 UTC m=+22.695431625" watchObservedRunningTime="2026-07-21 08:19:26.810372133 +0000 UTC m=+22.696042682" Jul 21 08:19:26.822324 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.822266 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ksjr8" podStartSLOduration=2.417575665 podStartE2EDuration="21.82225346s" totalImagesPullingTime="19.404677795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:26.821868167 +0000 UTC m=+22.707538715" watchObservedRunningTime="2026-07-21 08:19:26.82225346 +0000 UTC m=+22.707924008" Jul 21 08:19:26.833994 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.833955 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pfwh9" podStartSLOduration=6.990043056 podStartE2EDuration="21.83394287s" totalImagesPullingTime="14.843899814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:26.83350467 +0000 UTC m=+22.719175217" watchObservedRunningTime="2026-07-21 08:19:26.83394287 +0000 UTC m=+22.719613415" Jul 21 08:19:26.847791 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.847748 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tbtpj" podStartSLOduration=2.409582245 podStartE2EDuration="21.847735684s" totalImagesPullingTime="19.438153439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:26.847491068 +0000 UTC m=+22.733161628" watchObservedRunningTime="2026-07-21 08:19:26.847735684 +0000 UTC m=+22.733406231" Jul 21 08:19:26.859758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:26.859721 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bxvjt" podStartSLOduration=2.482243843 podStartE2EDuration="21.859708367s" totalImagesPullingTime="19.377464524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:26.859235684 +0000 UTC m=+22.744906233" watchObservedRunningTime="2026-07-21 08:19:26.859708367 +0000 UTC m=+22.745378914" Jul 21 08:19:27.668385 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.668174 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:27.668570 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.668174 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:27.668570 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:27.668500 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:27.668570 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:27.668547 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:27.791439 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.791413 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:19:27.792082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.791717 2553 generic.go:356] "Generic (PLEG): container finished" podID="b20b6770-0988-4982-9540-740ef9289986" containerID="704be4fb37470a232bfe403f8f037c904188bf97445f9f26d4fea8168330fddb" exitCode=1 Jul 21 08:19:27.792082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.791785 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"b51c2d9b90af60f7147cf80d8957bb1f437c98018c0c66ccdbcccc21c1f52413"} Jul 21 08:19:27.792082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.791858 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"01cc842416b6aa952f00e3feb21666c036d90ec86a3f2ef67002ab1c3beb3f4a"} Jul 21 08:19:27.792082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.791889 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"14e3fbcdd97a38e85dafbe2de89c433e4ba06165d56ec865ca65a9e3ba964902"} Jul 21 08:19:27.792082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.791919 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"ff728e92aab6decef84c8ef9068911c797c5e67062bf4d478d7d2ed08289828e"} Jul 21 08:19:27.792082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.791959 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerDied","Data":"704be4fb37470a232bfe403f8f037c904188bf97445f9f26d4fea8168330fddb"} Jul 21 08:19:27.793128 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.793106 2553 generic.go:356] "Generic (PLEG): container finished" podID="703d8cfe-167c-4860-b63d-517f2f57267e" containerID="6b9d043b8eac07308de6fd60bd6ad24d656a6142c9c26fb829e1ca0e648690da" exitCode=0 Jul 21 08:19:27.793236 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:27.793212 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerDied","Data":"6b9d043b8eac07308de6fd60bd6ad24d656a6142c9c26fb829e1ca0e648690da"} Jul 21 08:19:28.138727 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.138698 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 08:19:28.570891 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.570767 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T08:19:28.138722448Z","UUID":"949107d3-2203-4839-99bc-54fc91df6d32","Handler":null,"Name":"","Endpoint":""} Jul 21 08:19:28.572202 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.572177 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 08:19:28.572393 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.572226 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 08:19:28.670961 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.668410 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:28.670961 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:28.668546 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:28.797779 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.797739 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" event={"ID":"f835cf8c-f64a-49ca-bef2-2e2a5f84f458","Type":"ContainerStarted","Data":"79baeee3989d2f91b4cfba85ff2c7490f792f2a724b59827d2c8525fd0fe8da9"} Jul 21 08:19:28.799355 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.799327 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f7rrk" event={"ID":"3f87f794-2bb1-4ada-9342-48a583e8f7a9","Type":"ContainerStarted","Data":"6716dc68bf9717c820e0d6864106e052a385c6967b14d79a2e1f84ff9106e920"} Jul 21 08:19:28.813349 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:28.813273 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f7rrk" podStartSLOduration=4.433274075 podStartE2EDuration="23.813258672s" totalImagesPullingTime="19.379984597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:28.813099906 +0000 UTC m=+24.698770455" watchObservedRunningTime="2026-07-21 08:19:28.813258672 +0000 UTC m=+24.698929220" Jul 21 08:19:29.668635 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:29.668381 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:29.668812 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:29.668381 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:29.668812 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:29.668707 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:29.668812 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:29.668736 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:29.804857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:29.804819 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" event={"ID":"f835cf8c-f64a-49ca-bef2-2e2a5f84f458","Type":"ContainerStarted","Data":"e3eaede214a336ccbedd1c8efc9e6fcfb1e26cf8ac43cebad05e25af711b016f"} Jul 21 08:19:29.808276 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:29.808248 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:19:29.808796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:29.808765 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"2787b2c4a319a0ce1d3146799be5d979a395283a63377e2dbf1af4f4cf57131d"} Jul 21 08:19:29.823014 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:29.822956 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfzq9" podStartSLOduration=3.565324504 podStartE2EDuration="24.822936476s" totalImagesPullingTime="21.257611972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:29.821979783 +0000 UTC m=+25.707650332" watchObservedRunningTime="2026-07-21 08:19:29.822936476 +0000 UTC m=+25.708607026" Jul 21 08:19:30.668494 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:30.668464 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:30.668681 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:30.668602 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:31.668613 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:31.668538 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:31.669167 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:31.668545 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:31.669167 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:31.668664 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:31.669167 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:31.668729 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:31.768859 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:31.768820 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:31.769450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:31.769423 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:31.813411 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:31.813376 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:31.813939 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:31.813914 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pfwh9" Jul 21 08:19:32.668399 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.668378 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:32.668499 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:32.668486 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:32.819512 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.819234 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:19:32.820138 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.819829 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"3046cc927cc95d4d80a575225d6a65856f16b132977586e664552c8ad0e56b6e"} Jul 21 08:19:32.820256 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.820158 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:32.820256 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.820215 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:32.820256 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.820252 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:32.820512 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.820251 2553 scope.go:117] "RemoveContainer" containerID="704be4fb37470a232bfe403f8f037c904188bf97445f9f26d4fea8168330fddb" Jul 21 08:19:32.838272 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.838250 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:32.838398 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:32.838386 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:19:33.668679 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:33.668652 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:33.668787 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:33.668657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:33.668836 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:33.668781 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:33.668836 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:33.668826 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:33.825521 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:33.825496 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:19:33.825934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:33.825870 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" event={"ID":"b20b6770-0988-4982-9540-740ef9289986","Type":"ContainerStarted","Data":"90979651851fc1c40ca204a6226c3562e04ce7eb0c9d0fd5442148f87478500b"} Jul 21 08:19:33.827619 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:33.827592 2553 generic.go:356] "Generic (PLEG): container finished" podID="703d8cfe-167c-4860-b63d-517f2f57267e" containerID="4f750cade84b8e54c1332cfcea888b2c016ec66934bb542a619652d5fce889b2" exitCode=0 Jul 21 08:19:33.827716 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:33.827669 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerDied","Data":"4f750cade84b8e54c1332cfcea888b2c016ec66934bb542a619652d5fce889b2"} Jul 21 08:19:33.850368 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:33.850320 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" podStartSLOduration=9.375251118 podStartE2EDuration="28.850289886s" totalImagesPullingTime="19.475038768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:33.849194764 +0000 UTC m=+29.734865311" watchObservedRunningTime="2026-07-21 08:19:33.850289886 +0000 UTC m=+29.735960412" Jul 21 08:19:34.670120 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.669877 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:34.670265 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:34.670239 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:34.683325 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.683281 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-72jw7"] Jul 21 08:19:34.683484 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.683389 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:34.683547 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:34.683480 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:34.686214 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.686178 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9m59z"] Jul 21 08:19:34.695998 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.695975 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2ffz9"] Jul 21 08:19:34.696116 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.696059 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:34.696191 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:34.696172 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:34.832201 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.832169 2553 generic.go:356] "Generic (PLEG): container finished" podID="703d8cfe-167c-4860-b63d-517f2f57267e" containerID="895338cf55388e07e571e00c5c9297e1b150df2dd3522a41662c10617e0561a6" exitCode=0 Jul 21 08:19:34.832585 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.832255 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerDied","Data":"895338cf55388e07e571e00c5c9297e1b150df2dd3522a41662c10617e0561a6"} Jul 21 08:19:34.832720 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:34.832706 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:34.832838 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:34.832812 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:36.667754 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:36.667720 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:36.667754 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:36.667744 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:36.668443 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:36.667748 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:36.668443 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:36.667844 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:36.668443 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:36.667920 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:36.668443 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:36.668010 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:36.839473 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:36.839437 2553 generic.go:356] "Generic (PLEG): container finished" podID="703d8cfe-167c-4860-b63d-517f2f57267e" containerID="73950e96f4649fcd07dcf894100caab7fd43b49a396f77396cf7876fedb52e96" exitCode=0 Jul 21 08:19:36.839625 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:36.839479 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerDied","Data":"73950e96f4649fcd07dcf894100caab7fd43b49a396f77396cf7876fedb52e96"} Jul 21 08:19:38.271912 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:38.271878 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:38.272358 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.272124 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:38.272358 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.272203 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret podName:d8a24c11-99e9-4221-b50b-851f24298d13 nodeName:}" failed. No retries permitted until 2026-07-21 08:20:10.272182641 +0000 UTC m=+66.157853171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret") pod "global-pull-secret-syncer-2ffz9" (UID: "d8a24c11-99e9-4221-b50b-851f24298d13") : object "kube-system"/"original-pull-secret" not registered Jul 21 08:19:38.372867 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:38.372838 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:38.373053 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:38.372890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:38.373053 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.373037 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 08:19:38.373140 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.373063 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 08:19:38.373140 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.373076 2553 projected.go:196] Error preparing data for projected volume kube-api-access-6p27d for pod openshift-network-diagnostics/network-check-target-72jw7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:38.373140 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.373080 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:38.373140 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.373141 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:20:10.373121728 +0000 UTC m=+66.258792268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 08:19:38.373292 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.373155 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d podName:b8ebe912-4210-47a1-b6db-e301c388eb82 nodeName:}" failed. No retries permitted until 2026-07-21 08:20:10.37314817 +0000 UTC m=+66.258818695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6p27d" (UniqueName: "kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d") pod "network-check-target-72jw7" (UID: "b8ebe912-4210-47a1-b6db-e301c388eb82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 08:19:38.668582 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:38.668500 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:38.668753 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:38.668499 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:38.668753 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.668636 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jw7" podUID="b8ebe912-4210-47a1-b6db-e301c388eb82" Jul 21 08:19:38.668753 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.668710 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2ffz9" podUID="d8a24c11-99e9-4221-b50b-851f24298d13" Jul 21 08:19:38.668753 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:38.668502 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:38.668964 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:38.668791 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:19:39.484263 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.484192 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-134.ec2.internal" event="NodeReady" Jul 21 08:19:39.484713 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.484344 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 08:19:39.524933 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.524893 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6dw7h"] Jul 21 08:19:39.540980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.540598 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9g7hf"] Jul 21 08:19:39.540980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.540641 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.543410 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.543258 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 08:19:39.543410 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.543289 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw7kl\"" Jul 21 08:19:39.543723 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.543701 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 08:19:39.564097 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.564066 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6dw7h"] Jul 21 08:19:39.564250 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.564110 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9g7hf"] Jul 21 08:19:39.564250 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.564201 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:39.566694 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.566648 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 08:19:39.566694 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.566659 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 08:19:39.566877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.566788 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xmzts\"" Jul 21 08:19:39.566877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.566855 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 08:19:39.684040 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.683999 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de9de6-8a89-44ef-8c4c-291377d55d49-config-volume\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.684226 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.684040 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.684226 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.684117 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5l9\" (UniqueName: \"kubernetes.io/projected/b3de9de6-8a89-44ef-8c4c-291377d55d49-kube-api-access-wj5l9\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.684226 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.684139 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b529m\" (UniqueName: \"kubernetes.io/projected/14798834-275f-4627-a77c-d93602ef8a42-kube-api-access-b529m\") pod \"ingress-canary-9g7hf\" (UID: \"14798834-275f-4627-a77c-d93602ef8a42\") " pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:39.684226 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.684202 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14798834-275f-4627-a77c-d93602ef8a42-cert\") pod \"ingress-canary-9g7hf\" (UID: \"14798834-275f-4627-a77c-d93602ef8a42\") " pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:39.684464 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.684237 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3de9de6-8a89-44ef-8c4c-291377d55d49-tmp-dir\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.785417 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785326 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de9de6-8a89-44ef-8c4c-291377d55d49-config-volume\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.785417 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785376 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.785640 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785438 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5l9\" (UniqueName: \"kubernetes.io/projected/b3de9de6-8a89-44ef-8c4c-291377d55d49-kube-api-access-wj5l9\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.785640 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785462 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b529m\" (UniqueName: \"kubernetes.io/projected/14798834-275f-4627-a77c-d93602ef8a42-kube-api-access-b529m\") pod \"ingress-canary-9g7hf\" (UID: \"14798834-275f-4627-a77c-d93602ef8a42\") " pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:39.785640 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785481 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14798834-275f-4627-a77c-d93602ef8a42-cert\") pod \"ingress-canary-9g7hf\" (UID: \"14798834-275f-4627-a77c-d93602ef8a42\") " pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:39.785640 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785503 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3de9de6-8a89-44ef-8c4c-291377d55d49-tmp-dir\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.785640 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:39.785516 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:39.785640 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:39.785602 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:40.285579463 +0000 UTC m=+36.171249990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:19:39.785905 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785819 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3de9de6-8a89-44ef-8c4c-291377d55d49-tmp-dir\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.786048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.785992 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3de9de6-8a89-44ef-8c4c-291377d55d49-config-volume\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.791431 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.791405 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14798834-275f-4627-a77c-d93602ef8a42-cert\") pod \"ingress-canary-9g7hf\" (UID: \"14798834-275f-4627-a77c-d93602ef8a42\") " pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:39.798452 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.798425 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5l9\" (UniqueName: \"kubernetes.io/projected/b3de9de6-8a89-44ef-8c4c-291377d55d49-kube-api-access-wj5l9\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:39.798598 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.798518 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b529m\" (UniqueName: \"kubernetes.io/projected/14798834-275f-4627-a77c-d93602ef8a42-kube-api-access-b529m\") pod \"ingress-canary-9g7hf\" (UID: \"14798834-275f-4627-a77c-d93602ef8a42\") " pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:39.875030 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:39.875002 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9g7hf" Jul 21 08:19:40.289203 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.289167 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:40.289384 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:40.289295 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:40.289429 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:40.289385 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:41.289365994 +0000 UTC m=+37.175036520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:19:40.668103 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.668017 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:19:40.668593 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.668017 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:19:40.668593 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.668038 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:19:40.670573 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.670554 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 08:19:40.671789 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.671773 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 08:19:40.672109 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.671994 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 08:19:40.672109 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.672024 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 08:19:40.672109 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.672028 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bh656\"" Jul 21 08:19:40.672353 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:40.672026 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4qzvw\"" Jul 21 08:19:41.172764 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:41.172585 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9g7hf"] Jul 21 08:19:41.249400 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:41.249370 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14798834_275f_4627_a77c_d93602ef8a42.slice/crio-4ac68bbe1e6759842c6ec8ffa8904caf0db961f1a8a2ed85e7e63b13f84c22e0 WatchSource:0}: Error finding container 4ac68bbe1e6759842c6ec8ffa8904caf0db961f1a8a2ed85e7e63b13f84c22e0: Status 404 returned error can't find the container with id 4ac68bbe1e6759842c6ec8ffa8904caf0db961f1a8a2ed85e7e63b13f84c22e0 Jul 21 08:19:41.296916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:41.296891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:41.297039 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:41.297017 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:41.297121 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:41.297105 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:43.297082656 +0000 UTC m=+39.182753188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:19:41.852210 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:41.852170 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9g7hf" event={"ID":"14798834-275f-4627-a77c-d93602ef8a42","Type":"ContainerStarted","Data":"4ac68bbe1e6759842c6ec8ffa8904caf0db961f1a8a2ed85e7e63b13f84c22e0"} Jul 21 08:19:41.855724 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:41.855693 2553 generic.go:356] "Generic (PLEG): container finished" podID="703d8cfe-167c-4860-b63d-517f2f57267e" containerID="481596e34f1209ae27f6082adea64dc638ed4f1dad3fd0f9ed43547663d14c58" exitCode=0 Jul 21 08:19:41.855884 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:41.855733 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerDied","Data":"481596e34f1209ae27f6082adea64dc638ed4f1dad3fd0f9ed43547663d14c58"} Jul 21 08:19:42.861948 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:42.861691 2553 generic.go:356] "Generic (PLEG): container finished" podID="703d8cfe-167c-4860-b63d-517f2f57267e" containerID="e2ffffa440dc338b60ee6ee6e4b2d3fc25e4ad11d77923f002e2b4a1ac2ae026" exitCode=0 Jul 21 08:19:42.861948 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:42.861762 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerDied","Data":"e2ffffa440dc338b60ee6ee6e4b2d3fc25e4ad11d77923f002e2b4a1ac2ae026"} Jul 21 08:19:43.312298 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:43.312217 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:43.312457 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:43.312354 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:43.312457 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:43.312426 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:47.312407904 +0000 UTC m=+43.198078431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:19:43.868141 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:43.868099 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjksz" event={"ID":"703d8cfe-167c-4860-b63d-517f2f57267e","Type":"ContainerStarted","Data":"d4f3d328e5e26282ff857724a51f5495d5c7ece77647fad5ea25c1e86b3e92bf"} Jul 21 08:19:43.889126 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:43.889073 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mjksz" podStartSLOduration=8.225687772 podStartE2EDuration="38.889055335s" totalImagesPullingTime="30.663367563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:43.8872749 +0000 UTC m=+39.772945469" watchObservedRunningTime="2026-07-21 08:19:43.889055335 +0000 UTC m=+39.774725882" Jul 21 08:19:44.872193 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:44.872153 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9g7hf" event={"ID":"14798834-275f-4627-a77c-d93602ef8a42","Type":"ContainerStarted","Data":"19a12d9fa468d37a8aca09106c7cf3bd3baa93677e5bb76efe519edefc23156c"} Jul 21 08:19:44.887145 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:44.887084 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9g7hf" podStartSLOduration=3.240462365 podStartE2EDuration="5.887066332s" totalImagesPullingTime="2.646603967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:19:44.886614491 +0000 UTC m=+40.772285039" watchObservedRunningTime="2026-07-21 08:19:44.887066332 +0000 UTC m=+40.772736881" Jul 21 08:19:47.341864 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:47.341822 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:47.342243 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:47.341981 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:47.342243 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:47.342048 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:19:55.34203089 +0000 UTC m=+51.227701419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:19:55.401022 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:55.400977 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:19:55.401544 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:55.401118 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:19:55.401544 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:19:55.401181 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:20:11.401167736 +0000 UTC m=+67.286838263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:19:58.424177 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.424140 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp"] Jul 21 08:19:58.426469 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.426449 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.428614 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.428589 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 08:19:58.428769 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.428752 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 08:19:58.429550 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.429534 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 08:19:58.429596 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.429549 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 08:19:58.433919 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.433866 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp"] Jul 21 08:19:58.449295 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.449267 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs"] Jul 21 08:19:58.451399 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.451380 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.453575 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.453556 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 21 08:19:58.453688 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.453573 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 21 08:19:58.453688 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.453608 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 21 08:19:58.453776 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.453697 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 21 08:19:58.463013 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.462986 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs"] Jul 21 08:19:58.519041 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519003 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.519041 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksx24\" (UniqueName: \"kubernetes.io/projected/ffcffb11-478f-461b-ae9c-713b873d159c-kube-api-access-ksx24\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.519258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519059 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.519258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519160 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-ca\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.519258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519196 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffcffb11-478f-461b-ae9c-713b873d159c-tmp\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.519258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519213 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ffcffb11-478f-461b-ae9c-713b873d159c-klusterlet-config\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.519258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519230 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-hub\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.519258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519246 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/338368da-2b90-4ac5-b51a-6a4a47002e6a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.519541 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.519280 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgth4\" (UniqueName: \"kubernetes.io/projected/338368da-2b90-4ac5-b51a-6a4a47002e6a-kube-api-access-cgth4\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.619960 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.619919 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.619960 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.619962 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksx24\" (UniqueName: \"kubernetes.io/projected/ffcffb11-478f-461b-ae9c-713b873d159c-kube-api-access-ksx24\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.620210 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.619982 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.620210 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.620131 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-ca\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.620210 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.620167 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffcffb11-478f-461b-ae9c-713b873d159c-tmp\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.620210 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.620186 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ffcffb11-478f-461b-ae9c-713b873d159c-klusterlet-config\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.620210 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.620204 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-hub\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.620461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.620230 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/338368da-2b90-4ac5-b51a-6a4a47002e6a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.620461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.620283 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgth4\" (UniqueName: \"kubernetes.io/projected/338368da-2b90-4ac5-b51a-6a4a47002e6a-kube-api-access-cgth4\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.620616 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.620593 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ffcffb11-478f-461b-ae9c-713b873d159c-tmp\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.621060 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.621035 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/338368da-2b90-4ac5-b51a-6a4a47002e6a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.624551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.624523 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-ca\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.624666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.624533 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-hub\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.624666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.624558 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.624666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.624578 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/338368da-2b90-4ac5-b51a-6a4a47002e6a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.624666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.624585 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ffcffb11-478f-461b-ae9c-713b873d159c-klusterlet-config\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.628907 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.628871 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksx24\" (UniqueName: \"kubernetes.io/projected/ffcffb11-478f-461b-ae9c-713b873d159c-kube-api-access-ksx24\") pod \"klusterlet-addon-workmgr-56d789cd9f-tdpxp\" (UID: \"ffcffb11-478f-461b-ae9c-713b873d159c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.631182 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.631163 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgth4\" (UniqueName: \"kubernetes.io/projected/338368da-2b90-4ac5-b51a-6a4a47002e6a-kube-api-access-cgth4\") pod \"cluster-proxy-proxy-agent-84cbc4d498-4j8zs\" (UID: \"338368da-2b90-4ac5-b51a-6a4a47002e6a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.735895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.735808 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:19:58.771904 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.771865 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:19:58.871465 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.871422 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp"] Jul 21 08:19:58.875212 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:58.875184 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcffb11_478f_461b_ae9c_713b873d159c.slice/crio-1f2030ad2c45ab3f7541d79bb6530575ff498c67b786f35578062eb09eb3abbe WatchSource:0}: Error finding container 1f2030ad2c45ab3f7541d79bb6530575ff498c67b786f35578062eb09eb3abbe: Status 404 returned error can't find the container with id 1f2030ad2c45ab3f7541d79bb6530575ff498c67b786f35578062eb09eb3abbe Jul 21 08:19:58.904086 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.904057 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" event={"ID":"ffcffb11-478f-461b-ae9c-713b873d159c","Type":"ContainerStarted","Data":"1f2030ad2c45ab3f7541d79bb6530575ff498c67b786f35578062eb09eb3abbe"} Jul 21 08:19:58.911702 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:58.911670 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs"] Jul 21 08:19:58.915624 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:19:58.915600 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod338368da_2b90_4ac5_b51a_6a4a47002e6a.slice/crio-87faafb9eec52c456c962f8083e032d44d4cc94824a7d32a380675bd7fee9bc6 WatchSource:0}: Error finding container 87faafb9eec52c456c962f8083e032d44d4cc94824a7d32a380675bd7fee9bc6: Status 404 returned error can't find the container with id 87faafb9eec52c456c962f8083e032d44d4cc94824a7d32a380675bd7fee9bc6 Jul 21 08:19:59.910124 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:19:59.910049 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" event={"ID":"338368da-2b90-4ac5-b51a-6a4a47002e6a","Type":"ContainerStarted","Data":"87faafb9eec52c456c962f8083e032d44d4cc94824a7d32a380675bd7fee9bc6"} Jul 21 08:20:04.843432 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.843400 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bf86t" Jul 21 08:20:04.926297 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.926263 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" event={"ID":"ffcffb11-478f-461b-ae9c-713b873d159c","Type":"ContainerStarted","Data":"62b2dc502798c7bbd3ce9818c7f8cb45aee6154703f582fdd271048b6931013c"} Jul 21 08:20:04.926488 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.926468 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:20:04.928243 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.928218 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:20:04.928662 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.928642 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" event={"ID":"338368da-2b90-4ac5-b51a-6a4a47002e6a","Type":"ContainerStarted","Data":"63d1ec9e0d1c6b64fefc7be17d036e29687ca5255c2c285f0b78984970f50225"} Jul 21 08:20:04.928741 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.928678 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" event={"ID":"338368da-2b90-4ac5-b51a-6a4a47002e6a","Type":"ContainerStarted","Data":"86b0a5b88c50ad56ce26f8ce4d36a0a57e1f3bbcb77c4d6e439ac2ad3d660cab"} Jul 21 08:20:04.928741 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.928696 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" event={"ID":"338368da-2b90-4ac5-b51a-6a4a47002e6a","Type":"ContainerStarted","Data":"0564092382c8020dbd7edb8bcc11583dd0569a6c7b9cca7c662305e0107f2f2c"} Jul 21 08:20:04.944689 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.944637 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" podStartSLOduration=1.719062975 podStartE2EDuration="6.944618475s" totalImagesPullingTime="5.2255555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:20:04.944050862 +0000 UTC m=+60.829721410" watchObservedRunningTime="2026-07-21 08:20:04.944618475 +0000 UTC m=+60.830289023" Jul 21 08:20:04.975164 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:04.975116 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" podStartSLOduration=1.799414787 podStartE2EDuration="6.975101283s" totalImagesPullingTime="5.175686496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:20:04.974548747 +0000 UTC m=+60.860219296" watchObservedRunningTime="2026-07-21 08:20:04.975101283 +0000 UTC m=+60.860771829" Jul 21 08:20:10.310544 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.310502 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:20:10.312736 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.312713 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 08:20:10.323734 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.323713 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8a24c11-99e9-4221-b50b-851f24298d13-original-pull-secret\") pod \"global-pull-secret-syncer-2ffz9\" (UID: \"d8a24c11-99e9-4221-b50b-851f24298d13\") " pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:20:10.386955 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.386915 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2ffz9" Jul 21 08:20:10.411479 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.411445 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:20:10.411641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.411524 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:20:10.415271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.415243 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 08:20:10.415271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.415260 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 08:20:10.422386 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:20:10.422362 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 08:20:10.422493 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:20:10.422454 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:21:14.422430733 +0000 UTC m=+130.308101277 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : secret "metrics-daemon-secret" not found Jul 21 08:20:10.424612 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.424590 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 08:20:10.435811 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.435780 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p27d\" (UniqueName: \"kubernetes.io/projected/b8ebe912-4210-47a1-b6db-e301c388eb82-kube-api-access-6p27d\") pod \"network-check-target-72jw7\" (UID: \"b8ebe912-4210-47a1-b6db-e301c388eb82\") " pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:20:10.537130 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.537093 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2ffz9"] Jul 21 08:20:10.540363 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:20:10.540332 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a24c11_99e9_4221_b50b_851f24298d13.slice/crio-62121abf6bc3f3eae4ee5a680f1bcd567de6c0af05d0c73e7fb542f3760f0a81 WatchSource:0}: Error finding container 62121abf6bc3f3eae4ee5a680f1bcd567de6c0af05d0c73e7fb542f3760f0a81: Status 404 returned error can't find the container with id 62121abf6bc3f3eae4ee5a680f1bcd567de6c0af05d0c73e7fb542f3760f0a81 Jul 21 08:20:10.682781 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.682751 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4qzvw\"" Jul 21 08:20:10.690483 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.690456 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:20:10.837148 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.837118 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-72jw7"] Jul 21 08:20:10.840091 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:20:10.840067 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ebe912_4210_47a1_b6db_e301c388eb82.slice/crio-aeccf05d5c0f982f86cf8fa19d567e490aed947c232f789c617a00074da7ba7d WatchSource:0}: Error finding container aeccf05d5c0f982f86cf8fa19d567e490aed947c232f789c617a00074da7ba7d: Status 404 returned error can't find the container with id aeccf05d5c0f982f86cf8fa19d567e490aed947c232f789c617a00074da7ba7d Jul 21 08:20:10.944611 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.944525 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2ffz9" event={"ID":"d8a24c11-99e9-4221-b50b-851f24298d13","Type":"ContainerStarted","Data":"62121abf6bc3f3eae4ee5a680f1bcd567de6c0af05d0c73e7fb542f3760f0a81"} Jul 21 08:20:10.945708 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:10.945677 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-72jw7" event={"ID":"b8ebe912-4210-47a1-b6db-e301c388eb82","Type":"ContainerStarted","Data":"aeccf05d5c0f982f86cf8fa19d567e490aed947c232f789c617a00074da7ba7d"} Jul 21 08:20:11.418122 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:11.418049 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:20:11.418623 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:20:11.418243 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:20:11.418623 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:20:11.418349 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:20:43.418326182 +0000 UTC m=+99.303996725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:20:15.965554 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:15.965506 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2ffz9" event={"ID":"d8a24c11-99e9-4221-b50b-851f24298d13","Type":"ContainerStarted","Data":"1723cb3fea452b2b4e4f6d2eed0f41b6c2c1f145c39a2ce34c07357cd91aa38a"} Jul 21 08:20:15.967076 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:15.967055 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-72jw7" event={"ID":"b8ebe912-4210-47a1-b6db-e301c388eb82","Type":"ContainerStarted","Data":"2511b8a2de219dc8b46eef10b25e73c1fa1bc5436e6cba90b1b33ddf4f00af79"} Jul 21 08:20:15.967168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:15.967102 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:20:15.978867 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:15.978824 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2ffz9" podStartSLOduration=66.212944623 podStartE2EDuration="1m10.978810454s" totalImagesPullingTime="4.765865831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:20:15.977929905 +0000 UTC m=+71.863600453" watchObservedRunningTime="2026-07-21 08:20:15.978810454 +0000 UTC m=+71.864481037" Jul 21 08:20:15.991496 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:15.991450 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-72jw7" podStartSLOduration=66.528834201 podStartE2EDuration="1m10.99143735s" totalImagesPullingTime="4.462603149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:20:15.990391613 +0000 UTC m=+71.876062161" watchObservedRunningTime="2026-07-21 08:20:15.99143735 +0000 UTC m=+71.877107898" Jul 21 08:20:43.450837 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:43.450712 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:20:43.450837 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:20:43.450750 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:20:43.450837 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:20:43.450820 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:21:47.45080606 +0000 UTC m=+163.336476586 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:20:46.972599 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:20:46.972563 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-72jw7" Jul 21 08:21:14.471425 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:21:14.471368 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:21:14.471919 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:21:14.471526 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 08:21:14.471919 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:21:14.471604 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:23:16.471588018 +0000 UTC m=+252.357258545 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : secret "metrics-daemon-secret" not found Jul 21 08:21:42.552556 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:21:42.552496 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6dw7h" podUID="b3de9de6-8a89-44ef-8c4c-291377d55d49" Jul 21 08:21:43.211519 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:21:43.211490 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dw7h" Jul 21 08:21:43.693012 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:21:43.692957 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:21:47.506937 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:21:47.506898 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:21:47.507352 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:21:47.506907 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 08:21:47.507352 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:21:47.507046 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls podName:b3de9de6-8a89-44ef-8c4c-291377d55d49 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:49.507027798 +0000 UTC m=+285.392698324 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls") pod "dns-default-6dw7h" (UID: "b3de9de6-8a89-44ef-8c4c-291377d55d49") : secret "dns-default-metrics-tls" not found Jul 21 08:21:58.668874 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:21:58.668782 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:22:04.268848 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:22:04.268777 2553 generic.go:356] "Generic (PLEG): container finished" podID="ffcffb11-478f-461b-ae9c-713b873d159c" containerID="62b2dc502798c7bbd3ce9818c7f8cb45aee6154703f582fdd271048b6931013c" exitCode=1 Jul 21 08:22:04.269241 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:22:04.268854 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" event={"ID":"ffcffb11-478f-461b-ae9c-713b873d159c","Type":"ContainerDied","Data":"62b2dc502798c7bbd3ce9818c7f8cb45aee6154703f582fdd271048b6931013c"} Jul 21 08:22:04.269241 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:22:04.269196 2553 scope.go:117] "RemoveContainer" containerID="62b2dc502798c7bbd3ce9818c7f8cb45aee6154703f582fdd271048b6931013c" Jul 21 08:22:04.926449 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:22:04.926416 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:22:05.274042 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:22:05.273957 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" event={"ID":"ffcffb11-478f-461b-ae9c-713b873d159c","Type":"ContainerStarted","Data":"530d20dc44851c986745d399fcad08f00af37339e4971cbf494be5ede23b3555"} Jul 21 08:22:05.274510 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:22:05.274196 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:22:05.274826 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:22:05.274809 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56d789cd9f-tdpxp" Jul 21 08:23:11.639804 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.639770 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tg755"] Jul 21 08:23:11.641901 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.641884 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.644471 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:11.644434 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" Jul 21 08:23:11.644587 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:11.644536 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-tg755\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="55a3d257-5301-44c0-8222-9acecc1fdf19" pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.644587 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:11.644528 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" Jul 21 08:23:11.644587 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:11.644514 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 21 08:23:11.644954 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:11.644934 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 21 08:23:11.645942 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:11.645923 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-gbchk\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbchk\"" type="*v1.Secret" Jul 21 08:23:11.663524 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.663501 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tg755"] Jul 21 08:23:11.794982 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.794926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55a3d257-5301-44c0-8222-9acecc1fdf19-crio-socket\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.794982 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.794984 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55a3d257-5301-44c0-8222-9acecc1fdf19-data-volume\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.795223 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.795008 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.795223 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.795074 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.795223 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.795096 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wbf\" (UniqueName: \"kubernetes.io/projected/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-api-access-x5wbf\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.896536 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.896439 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55a3d257-5301-44c0-8222-9acecc1fdf19-crio-socket\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.896536 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.896488 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55a3d257-5301-44c0-8222-9acecc1fdf19-data-volume\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.896536 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.896506 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.896755 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.896573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55a3d257-5301-44c0-8222-9acecc1fdf19-crio-socket\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.896755 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.896580 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.896755 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.896628 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wbf\" (UniqueName: \"kubernetes.io/projected/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-api-access-x5wbf\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:11.896848 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:11.896814 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55a3d257-5301-44c0-8222-9acecc1fdf19-data-volume\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:12.658500 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:12.658464 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 08:23:12.897570 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:12.897531 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition Jul 21 08:23:12.897739 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:12.897547 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: failed to sync configmap cache: timed out waiting for the condition Jul 21 08:23:12.897739 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:12.897625 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:13.39760609 +0000 UTC m=+249.283276617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : failed to sync secret cache: timed out waiting for the condition Jul 21 08:23:12.897739 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:12.897710 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:13.397694175 +0000 UTC m=+249.283364701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : failed to sync configmap cache: timed out waiting for the condition Jul 21 08:23:12.902415 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:12.902386 2553 projected.go:291] Couldn't get configMap openshift-insights/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 21 08:23:12.902551 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:12.902423 2553 projected.go:196] Error preparing data for projected volume kube-api-access-x5wbf for pod openshift-insights/insights-runtime-extractor-tg755: failed to sync configmap cache: timed out waiting for the condition Jul 21 08:23:12.902551 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:12.902478 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-api-access-x5wbf podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:13.402465058 +0000 UTC m=+249.288135585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x5wbf" (UniqueName: "kubernetes.io/projected/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-api-access-x5wbf") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : failed to sync configmap cache: timed out waiting for the condition Jul 21 08:23:12.966718 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:12.966694 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 08:23:13.040878 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.040844 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 08:23:13.096075 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.096043 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbchk\"" Jul 21 08:23:13.188342 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.188295 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 08:23:13.408024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.407942 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:13.408024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.407990 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:13.408024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.408010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wbf\" (UniqueName: \"kubernetes.io/projected/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-api-access-x5wbf\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:13.408264 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:13.408092 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:13.408264 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:13.408102 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:13.408264 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:13.408168 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:14.408147987 +0000 UTC m=+250.293818513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : configmap "kube-rbac-proxy" not found Jul 21 08:23:13.408264 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:13.408188 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:14.408179547 +0000 UTC m=+250.293850072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:13.410444 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.410413 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wbf\" (UniqueName: \"kubernetes.io/projected/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-api-access-x5wbf\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:13.599488 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:13.599465 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksjr8_d775c285-f2a3-42b5-8867-88a0faefff5f/dns-node-resolver/0.log" Jul 21 08:23:14.400375 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:14.400350 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxvjt_f2090af5-b4b4-457b-8a32-8e361407a9ea/node-ca/0.log" Jul 21 08:23:14.414806 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:14.414776 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:14.414969 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:14.414823 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:14.414969 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:14.414902 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:14.414969 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:14.414928 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:14.414969 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:14.414962 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:16.414946969 +0000 UTC m=+252.300617494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : configmap "kube-rbac-proxy" not found Jul 21 08:23:14.415109 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:14.414976 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:16.414970615 +0000 UTC m=+252.300641142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:15.201171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:15.201144 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9g7hf_14798834-275f-4627-a77c-d93602ef8a42/serve-healthcheck-canary/0.log" Jul 21 08:23:16.430797 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:16.430762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:16.431173 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:16.430809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:16.431173 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:16.430888 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:16.431173 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:16.430942 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:16.431173 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:16.430955 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:20.430938315 +0000 UTC m=+256.316608841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : configmap "kube-rbac-proxy" not found Jul 21 08:23:16.431173 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:16.430982 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:20.430971139 +0000 UTC m=+256.316641664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:16.532086 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:16.532048 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:23:16.532257 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:16.532220 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 08:23:16.532375 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:16.532299 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs podName:0bcdb240-7d2d-45c8-b455-e65e7c28a22a nodeName:}" failed. No retries permitted until 2026-07-21 08:25:18.532278295 +0000 UTC m=+374.417948826 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs") pod "network-metrics-daemon-9m59z" (UID: "0bcdb240-7d2d-45c8-b455-e65e7c28a22a") : secret "metrics-daemon-secret" not found Jul 21 08:23:20.458551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:20.458512 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:20.459060 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:20.458564 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:20.459060 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:20.458643 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 08:23:20.459060 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:20.458668 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 08:23:20.459060 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:20.458717 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:28.458702073 +0000 UTC m=+264.344372600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : secret "insights-runtime-extractor-tls" not found Jul 21 08:23:20.459060 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:20.458730 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm podName:55a3d257-5301-44c0-8222-9acecc1fdf19 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:28.458724389 +0000 UTC m=+264.344394914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tg755" (UID: "55a3d257-5301-44c0-8222-9acecc1fdf19") : configmap "kube-rbac-proxy" not found Jul 21 08:23:28.520412 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:28.520297 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:28.520855 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:28.520412 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:28.521052 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:28.521031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55a3d257-5301-44c0-8222-9acecc1fdf19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:28.522666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:28.522642 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55a3d257-5301-44c0-8222-9acecc1fdf19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tg755\" (UID: \"55a3d257-5301-44c0-8222-9acecc1fdf19\") " pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:28.750490 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:28.750454 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tg755" Jul 21 08:23:28.881554 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:28.881512 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tg755"] Jul 21 08:23:28.884710 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:23:28.884685 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a3d257_5301_44c0_8222_9acecc1fdf19.slice/crio-7a50de1186b72ce87aaa07d4a8d289d52cea48e4a82a8b7f96c83f17c17c5e78 WatchSource:0}: Error finding container 7a50de1186b72ce87aaa07d4a8d289d52cea48e4a82a8b7f96c83f17c17c5e78: Status 404 returned error can't find the container with id 7a50de1186b72ce87aaa07d4a8d289d52cea48e4a82a8b7f96c83f17c17c5e78 Jul 21 08:23:29.514893 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:29.514800 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg755" event={"ID":"55a3d257-5301-44c0-8222-9acecc1fdf19","Type":"ContainerStarted","Data":"7a50de1186b72ce87aaa07d4a8d289d52cea48e4a82a8b7f96c83f17c17c5e78"} Jul 21 08:23:34.532846 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:34.532805 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg755" event={"ID":"55a3d257-5301-44c0-8222-9acecc1fdf19","Type":"ContainerStarted","Data":"0fedaf63d7c4686b90618e2731115c2f36c9168d2df8b0ed0498af3314984d6f"} Jul 21 08:23:37.543928 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:37.543846 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg755" event={"ID":"55a3d257-5301-44c0-8222-9acecc1fdf19","Type":"ContainerStarted","Data":"292d0178751d6e5dabceacbdb0b94ccf18f773e8acdce4642ca015b72fc37fd7"} Jul 21 08:23:41.559127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:41.559091 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tg755" event={"ID":"55a3d257-5301-44c0-8222-9acecc1fdf19","Type":"ContainerStarted","Data":"7d499552b4ff6ce67c41aefb53e493d0c527d5531a2d9718542b04523c1ac1cc"} Jul 21 08:23:41.580035 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:41.579982 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tg755" podStartSLOduration=18.871395351 podStartE2EDuration="30.579968587s" totalImagesPullingTime="11.708573236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 08:23:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:41.578662515 +0000 UTC m=+277.464333074" watchObservedRunningTime="2026-07-21 08:23:41.579968587 +0000 UTC m=+277.465639135" Jul 21 08:23:46.212759 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:46.212717 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6dw7h" podUID="b3de9de6-8a89-44ef-8c4c-291377d55d49" Jul 21 08:23:46.574933 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:46.574858 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dw7h" Jul 21 08:23:49.576860 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:49.576769 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:23:49.579261 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:49.579236 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3de9de6-8a89-44ef-8c4c-291377d55d49-metrics-tls\") pod \"dns-default-6dw7h\" (UID: \"b3de9de6-8a89-44ef-8c4c-291377d55d49\") " pod="openshift-dns/dns-default-6dw7h" Jul 21 08:23:49.878255 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:49.878180 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jw7kl\"" Jul 21 08:23:49.886267 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:49.886248 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dw7h" Jul 21 08:23:50.022142 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:50.022109 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6dw7h"] Jul 21 08:23:50.025352 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:23:50.025328 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3de9de6_8a89_44ef_8c4c_291377d55d49.slice/crio-9cf5534ed3436aa23567d5c4e156f7f99daf30c0af30fb9253782293f0db7f33 WatchSource:0}: Error finding container 9cf5534ed3436aa23567d5c4e156f7f99daf30c0af30fb9253782293f0db7f33: Status 404 returned error can't find the container with id 9cf5534ed3436aa23567d5c4e156f7f99daf30c0af30fb9253782293f0db7f33 Jul 21 08:23:50.588185 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:50.588148 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dw7h" event={"ID":"b3de9de6-8a89-44ef-8c4c-291377d55d49","Type":"ContainerStarted","Data":"9cf5534ed3436aa23567d5c4e156f7f99daf30c0af30fb9253782293f0db7f33"} Jul 21 08:23:52.120332 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.120276 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nkdwl"] Jul 21 08:23:52.122355 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.122338 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.126486 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.126461 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 08:23:52.126624 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.126604 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 08:23:52.126825 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.126810 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 08:23:52.126901 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.126836 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 08:23:52.126958 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.126837 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 08:23:52.127925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.127906 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 08:23:52.128463 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.128442 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x4bjv\"" Jul 21 08:23:52.196018 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.195983 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gc96\" (UniqueName: \"kubernetes.io/projected/3d769311-4e69-43c3-95a4-1be71d1bbe14-kube-api-access-6gc96\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196018 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196020 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-tls\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196283 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196045 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d769311-4e69-43c3-95a4-1be71d1bbe14-metrics-client-ca\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196283 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196143 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196283 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196211 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-sys\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196283 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196243 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-textfile\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196288 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196351 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-root\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.196461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.196370 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-wtmp\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297338 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297267 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-sys\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297338 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297343 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-textfile\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297371 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297386 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-sys\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297404 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-root\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297448 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-wtmp\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297478 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gc96\" (UniqueName: \"kubernetes.io/projected/3d769311-4e69-43c3-95a4-1be71d1bbe14-kube-api-access-6gc96\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297500 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-tls\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297519 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d769311-4e69-43c3-95a4-1be71d1bbe14-metrics-client-ca\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297550 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297961 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297546 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-root\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297961 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:52.297642 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 08:23:52.297961 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297650 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-wtmp\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.297961 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:52.297726 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-tls podName:3d769311-4e69-43c3-95a4-1be71d1bbe14 nodeName:}" failed. No retries permitted until 2026-07-21 08:23:52.797703648 +0000 UTC m=+288.683374360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-tls") pod "node-exporter-nkdwl" (UID: "3d769311-4e69-43c3-95a4-1be71d1bbe14") : secret "node-exporter-tls" not found Jul 21 08:23:52.297961 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.297738 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-textfile\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.298201 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.298097 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.298201 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.298192 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d769311-4e69-43c3-95a4-1be71d1bbe14-metrics-client-ca\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.299942 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.299925 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.310961 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.310934 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gc96\" (UniqueName: \"kubernetes.io/projected/3d769311-4e69-43c3-95a4-1be71d1bbe14-kube-api-access-6gc96\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.596095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.596061 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dw7h" event={"ID":"b3de9de6-8a89-44ef-8c4c-291377d55d49","Type":"ContainerStarted","Data":"6e25a219b840bde69c605e061f737f0e301d3b1480c6624c32c5abacd82af3a9"} Jul 21 08:23:52.596278 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.596119 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dw7h" event={"ID":"b3de9de6-8a89-44ef-8c4c-291377d55d49","Type":"ContainerStarted","Data":"4e5494a42de05ad04738260c19de75284cf89ddde45315c62f88c3a55706a8b8"} Jul 21 08:23:52.596278 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.596167 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6dw7h" Jul 21 08:23:52.614379 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.614304 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6dw7h" podStartSLOduration=252.021544562 podStartE2EDuration="4m13.614289045s" totalImagesPullingTime="1.592744483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:52.613411739 +0000 UTC m=+288.499082311" watchObservedRunningTime="2026-07-21 08:23:52.614289045 +0000 UTC m=+288.499959592" Jul 21 08:23:52.800721 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.800672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-tls\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:52.803162 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:52.803137 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3d769311-4e69-43c3-95a4-1be71d1bbe14-node-exporter-tls\") pod \"node-exporter-nkdwl\" (UID: \"3d769311-4e69-43c3-95a4-1be71d1bbe14\") " pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:53.031513 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:53.031481 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nkdwl" Jul 21 08:23:53.041367 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:23:53.041341 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d769311_4e69_43c3_95a4_1be71d1bbe14.slice/crio-a4b889a66752ea76297ee74f26ecfc08ab79436bd72e05a116643d20934fe5fd WatchSource:0}: Error finding container a4b889a66752ea76297ee74f26ecfc08ab79436bd72e05a116643d20934fe5fd: Status 404 returned error can't find the container with id a4b889a66752ea76297ee74f26ecfc08ab79436bd72e05a116643d20934fe5fd Jul 21 08:23:53.600667 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:53.600633 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkdwl" event={"ID":"3d769311-4e69-43c3-95a4-1be71d1bbe14","Type":"ContainerStarted","Data":"a4b889a66752ea76297ee74f26ecfc08ab79436bd72e05a116643d20934fe5fd"} Jul 21 08:23:54.606394 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:54.606355 2553 generic.go:356] "Generic (PLEG): container finished" podID="3d769311-4e69-43c3-95a4-1be71d1bbe14" containerID="5dc2d38c198a336486bf022345c4bbd9c4088c375487420ce6ae1e71210e415f" exitCode=0 Jul 21 08:23:54.606794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:54.606438 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkdwl" event={"ID":"3d769311-4e69-43c3-95a4-1be71d1bbe14","Type":"ContainerDied","Data":"5dc2d38c198a336486bf022345c4bbd9c4088c375487420ce6ae1e71210e415f"} Jul 21 08:23:55.612430 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:55.612396 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkdwl" event={"ID":"3d769311-4e69-43c3-95a4-1be71d1bbe14","Type":"ContainerStarted","Data":"5aa9a4b4509070063bdbdfbaa126bab7ea8784d2277490c8b73fb2697cd2b839"} Jul 21 08:23:55.612802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:55.612472 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkdwl" event={"ID":"3d769311-4e69-43c3-95a4-1be71d1bbe14","Type":"ContainerStarted","Data":"74ad48054139d376b7b28b6b2a7c125d4950f25dac16c421b08cfe8b50ca7436"} Jul 21 08:23:55.639544 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:55.639492 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nkdwl" podStartSLOduration=2.680986864 podStartE2EDuration="3.63947596s" totalImagesPullingTime="887.330122ms" totalInitContainerRuntime="71.158974ms" isStatefulPod=true podCreationTimestamp="2026-07-21 08:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:55.639134445 +0000 UTC m=+291.524805005" watchObservedRunningTime="2026-07-21 08:23:55.63947596 +0000 UTC m=+291.525146508" Jul 21 08:23:56.256041 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.256006 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6688d859f-s89c8"] Jul 21 08:23:56.258686 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.258670 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.262263 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.262244 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 08:23:56.262683 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.262668 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5cf6fdn20o5j6\"" Jul 21 08:23:56.262834 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.262818 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 08:23:56.263173 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.263156 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xgmlz\"" Jul 21 08:23:56.263744 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.263728 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 08:23:56.263808 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.263732 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 08:23:56.263808 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.263764 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 08:23:56.277787 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.277762 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6688d859f-s89c8"] Jul 21 08:23:56.326057 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326021 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.326057 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326061 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a85ff1a4-22d9-44f0-be89-bc146b606b19-metrics-client-ca\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.326335 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326086 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-grpc-tls\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.326335 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326159 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.326335 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326192 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.326335 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326274 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.326524 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326363 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-tls\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.326524 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.326402 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmtsb\" (UniqueName: \"kubernetes.io/projected/a85ff1a4-22d9-44f0-be89-bc146b606b19-kube-api-access-lmtsb\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427111 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427068 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427279 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427121 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-tls\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427279 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427141 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmtsb\" (UniqueName: \"kubernetes.io/projected/a85ff1a4-22d9-44f0-be89-bc146b606b19-kube-api-access-lmtsb\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427279 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427173 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427279 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427196 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a85ff1a4-22d9-44f0-be89-bc146b606b19-metrics-client-ca\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427279 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427216 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-grpc-tls\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427279 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.427279 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.427271 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.428089 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.428054 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a85ff1a4-22d9-44f0-be89-bc146b606b19-metrics-client-ca\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.430644 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.430623 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.430740 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.430700 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-tls\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.430819 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.430792 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.430913 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.430870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-grpc-tls\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.431074 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.431055 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.431109 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.431055 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a85ff1a4-22d9-44f0-be89-bc146b606b19-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.450203 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.450177 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmtsb\" (UniqueName: \"kubernetes.io/projected/a85ff1a4-22d9-44f0-be89-bc146b606b19-kube-api-access-lmtsb\") pod \"thanos-querier-6688d859f-s89c8\" (UID: \"a85ff1a4-22d9-44f0-be89-bc146b606b19\") " pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.567565 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.567474 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:23:56.711481 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.711445 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6688d859f-s89c8"] Jul 21 08:23:56.714590 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:23:56.714561 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85ff1a4_22d9_44f0_be89_bc146b606b19.slice/crio-4693f6e23b7c0d4bea4d9991cf8108daba0f56a009afdeecbf38998271ae5c8e WatchSource:0}: Error finding container 4693f6e23b7c0d4bea4d9991cf8108daba0f56a009afdeecbf38998271ae5c8e: Status 404 returned error can't find the container with id 4693f6e23b7c0d4bea4d9991cf8108daba0f56a009afdeecbf38998271ae5c8e Jul 21 08:23:56.901618 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.901541 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth"] Jul 21 08:23:56.904261 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.904242 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:56.907157 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.907137 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x66rg\"" Jul 21 08:23:56.907243 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.907135 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 08:23:56.916322 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:56.916283 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth"] Jul 21 08:23:57.034230 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.034181 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ece16ad-237d-4d25-bf60-12faf52b229e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttvth\" (UID: \"0ece16ad-237d-4d25-bf60-12faf52b229e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.034230 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.034231 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcl74\" (UniqueName: \"kubernetes.io/projected/0ece16ad-237d-4d25-bf60-12faf52b229e-kube-api-access-jcl74\") pod \"monitoring-plugin-7b7b7bc8bf-ttvth\" (UID: \"0ece16ad-237d-4d25-bf60-12faf52b229e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.134962 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.134929 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ece16ad-237d-4d25-bf60-12faf52b229e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttvth\" (UID: \"0ece16ad-237d-4d25-bf60-12faf52b229e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.134962 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.134965 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcl74\" (UniqueName: \"kubernetes.io/projected/0ece16ad-237d-4d25-bf60-12faf52b229e-kube-api-access-jcl74\") pod \"monitoring-plugin-7b7b7bc8bf-ttvth\" (UID: \"0ece16ad-237d-4d25-bf60-12faf52b229e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.135171 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:57.135078 2553 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 21 08:23:57.135171 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:23:57.135138 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ece16ad-237d-4d25-bf60-12faf52b229e-monitoring-plugin-cert podName:0ece16ad-237d-4d25-bf60-12faf52b229e nodeName:}" failed. No retries permitted until 2026-07-21 08:23:57.635122576 +0000 UTC m=+293.520793101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0ece16ad-237d-4d25-bf60-12faf52b229e-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-ttvth" (UID: "0ece16ad-237d-4d25-bf60-12faf52b229e") : secret "monitoring-plugin-cert" not found Jul 21 08:23:57.146436 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.146416 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcl74\" (UniqueName: \"kubernetes.io/projected/0ece16ad-237d-4d25-bf60-12faf52b229e-kube-api-access-jcl74\") pod \"monitoring-plugin-7b7b7bc8bf-ttvth\" (UID: \"0ece16ad-237d-4d25-bf60-12faf52b229e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.624116 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.624079 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" event={"ID":"a85ff1a4-22d9-44f0-be89-bc146b606b19","Type":"ContainerStarted","Data":"4693f6e23b7c0d4bea4d9991cf8108daba0f56a009afdeecbf38998271ae5c8e"} Jul 21 08:23:57.639604 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.639563 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ece16ad-237d-4d25-bf60-12faf52b229e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttvth\" (UID: \"0ece16ad-237d-4d25-bf60-12faf52b229e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.642616 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.642590 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0ece16ad-237d-4d25-bf60-12faf52b229e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ttvth\" (UID: \"0ece16ad-237d-4d25-bf60-12faf52b229e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.813727 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.813695 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:57.964402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:57.964372 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth"] Jul 21 08:23:57.967396 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:23:57.967367 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ece16ad_237d_4d25_bf60_12faf52b229e.slice/crio-56373d8710b08b81f0382a148fda45bb36976cf74e87c50802ad1d71f611c70c WatchSource:0}: Error finding container 56373d8710b08b81f0382a148fda45bb36976cf74e87c50802ad1d71f611c70c: Status 404 returned error can't find the container with id 56373d8710b08b81f0382a148fda45bb36976cf74e87c50802ad1d71f611c70c Jul 21 08:23:58.448912 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.448882 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:23:58.452237 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.452209 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.456573 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.456548 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 08:23:58.456881 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.456668 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 08:23:58.457285 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.457266 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 08:23:58.457487 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.457439 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 08:23:58.457826 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.457755 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 08:23:58.458240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.458213 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 08:23:58.458412 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.458219 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 08:23:58.459710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.459685 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d525j\"" Jul 21 08:23:58.459987 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.459782 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 08:23:58.460925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.460292 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 08:23:58.460925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.460457 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 08:23:58.460925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.460473 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 08:23:58.460925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.460606 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 08:23:58.460925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.460825 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 08:23:58.460925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.460923 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-73b63larn9eoe\"" Jul 21 08:23:58.471619 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.471591 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 08:23:58.471778 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.471589 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 08:23:58.477730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.477668 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:23:58.547193 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547158 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547193 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547192 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547213 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547238 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547344 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547390 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547419 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547446 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547572 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547598 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547625 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547643 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547665 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.547730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547689 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2754m\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-kube-api-access-2754m\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.548008 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547735 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.548008 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547769 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.548008 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547806 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-config\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.548008 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547848 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.548008 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.547875 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.630994 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.630924 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" event={"ID":"0ece16ad-237d-4d25-bf60-12faf52b229e","Type":"ContainerStarted","Data":"56373d8710b08b81f0382a148fda45bb36976cf74e87c50802ad1d71f611c70c"} Jul 21 08:23:58.649421 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649386 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649608 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649434 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649608 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649459 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649608 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649482 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649608 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649524 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649608 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649563 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649608 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649587 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649619 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649648 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649677 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649701 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2754m\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-kube-api-access-2754m\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649730 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649787 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-config\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649859 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649900 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.649925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649921 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.650493 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649946 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.650493 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.649972 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.651065 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.650650 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.651543 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.651506 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.651806 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.651737 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.652373 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.652347 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.652604 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.652578 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.653898 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.653875 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.655777 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.655748 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.656228 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.656202 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.656967 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.656706 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.656967 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.656932 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.657479 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.657455 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.658398 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.658007 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.658398 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.658017 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-config\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.658398 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.658020 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.658831 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.658789 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.658950 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.658932 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.659149 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.659123 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.659828 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.659785 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.659980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.659961 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.662713 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.662691 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2754m\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-kube-api-access-2754m\") pod \"prometheus-k8s-0\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:58.773000 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:58.772906 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:23:59.168869 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.168833 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:23:59.394386 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:23:59.394251 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972e65ff_3913_4f02_9195_04bf617ef9a8.slice/crio-b45a8b184dae106087cd13fd07cbdb252838c61c036889d8b5179dcfdb035758 WatchSource:0}: Error finding container b45a8b184dae106087cd13fd07cbdb252838c61c036889d8b5179dcfdb035758: Status 404 returned error can't find the container with id b45a8b184dae106087cd13fd07cbdb252838c61c036889d8b5179dcfdb035758 Jul 21 08:23:59.635916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.635882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"b45a8b184dae106087cd13fd07cbdb252838c61c036889d8b5179dcfdb035758"} Jul 21 08:23:59.637382 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.637352 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" event={"ID":"0ece16ad-237d-4d25-bf60-12faf52b229e","Type":"ContainerStarted","Data":"d69654045eb0e950b6fb995343b59f0774b49b10bdd8606d90ab754626571e52"} Jul 21 08:23:59.637536 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.637519 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:59.639587 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.639562 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" event={"ID":"a85ff1a4-22d9-44f0-be89-bc146b606b19","Type":"ContainerStarted","Data":"3bf19cb207b8f7c72598971fc01bdf778bb7596326cb29e96adc27cf2eeae379"} Jul 21 08:23:59.639701 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.639611 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" event={"ID":"a85ff1a4-22d9-44f0-be89-bc146b606b19","Type":"ContainerStarted","Data":"2ffad06b4a35004b9f6a6a9af0219a42eb96005d0b3f71a6fe9de19b0aa6982a"} Jul 21 08:23:59.639701 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.639638 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" event={"ID":"a85ff1a4-22d9-44f0-be89-bc146b606b19","Type":"ContainerStarted","Data":"580f857ebffe707723134341f343a1c9af3d5975e2b71d5667bf36ec08e15d4e"} Jul 21 08:23:59.643261 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.643243 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" Jul 21 08:23:59.654715 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:23:59.654622 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ttvth" podStartSLOduration=2.197652166 podStartE2EDuration="3.654606742s" totalImagesPullingTime="1.456954576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:23:59.653485724 +0000 UTC m=+295.539156307" watchObservedRunningTime="2026-07-21 08:23:59.654606742 +0000 UTC m=+295.540277280" Jul 21 08:24:00.652908 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:00.652872 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"e9ff2800e568d1bdc82dd56a317554b185e39bd1d57a70f449fdc9708472d539"} Jul 21 08:24:00.655972 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:00.655942 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" event={"ID":"a85ff1a4-22d9-44f0-be89-bc146b606b19","Type":"ContainerStarted","Data":"2303581409ac7137becbeb2b7cbf2e0ee2955d3c3719c57268a219ff1d7c7331"} Jul 21 08:24:01.662746 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:01.662706 2553 generic.go:356] "Generic (PLEG): container finished" podID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerID="e9ff2800e568d1bdc82dd56a317554b185e39bd1d57a70f449fdc9708472d539" exitCode=0 Jul 21 08:24:01.663151 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:01.662797 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"e9ff2800e568d1bdc82dd56a317554b185e39bd1d57a70f449fdc9708472d539"} Jul 21 08:24:01.665775 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:01.665752 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" event={"ID":"a85ff1a4-22d9-44f0-be89-bc146b606b19","Type":"ContainerStarted","Data":"a15eec257b56dc761d47b9b6ed9e590a0ea87d7c1b6e40eb9db9951ba2ce3923"} Jul 21 08:24:01.665888 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:01.665819 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" event={"ID":"a85ff1a4-22d9-44f0-be89-bc146b606b19","Type":"ContainerStarted","Data":"c0a98dae93a49f17a8fca1220cf5eaece84de8abc70b3c3b6ddc5520c16d2078"} Jul 21 08:24:01.670154 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:24:01.670133 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9m59z" podUID="0bcdb240-7d2d-45c8-b455-e65e7c28a22a" Jul 21 08:24:01.707518 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:01.707469 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" podStartSLOduration=2.350357204 podStartE2EDuration="5.70745335s" totalImagesPullingTime="3.357096146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:24:01.705767342 +0000 UTC m=+297.591437900" watchObservedRunningTime="2026-07-21 08:24:01.70745335 +0000 UTC m=+297.593123897" Jul 21 08:24:02.603863 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:02.603833 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6dw7h" Jul 21 08:24:02.672612 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:02.672578 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:24:04.914428 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:04.914397 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:24:04.915539 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:04.915117 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:24:04.920651 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:04.920627 2553 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 08:24:05.685638 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:05.685601 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"6dce055f2ba4277dd85b32ab5ca86d9fb075fed61d0cedbc83cc9b2995df3cb4"} Jul 21 08:24:05.685835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:05.685674 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"353b68750f036b77cd569bdc5a3d080a2ac376aab5e288a7b00a335896cff07d"} Jul 21 08:24:05.685835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:05.685720 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"3076447cdbc9bd5e7d26d8d75ad50bcfbd008fadef51ff135f8361dde3c69153"} Jul 21 08:24:05.685835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:05.685755 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"6fc1b7cd3a8fb0967f2ae17be363f94e7b1568a8575bd09902e421e063cf360a"} Jul 21 08:24:05.685835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:05.685791 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"dd377287baa50eaea60f7fceb9a3902fa5d1fd9f190f44770aaf89766b818092"} Jul 21 08:24:05.685835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:05.685832 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerStarted","Data":"1aad9c40b06b46287defea1fc8b29fc2fba45199c4fcfd71200841636122aa1d"} Jul 21 08:24:05.750064 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:05.746593 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.290294024 podStartE2EDuration="7.746572551s" totalImagesPullingTime="4.432453473s" totalInitContainerRuntime="23.825054ms" isStatefulPod=false podCreationTimestamp="2026-07-21 08:23:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:24:05.74592286 +0000 UTC m=+301.631593391" watchObservedRunningTime="2026-07-21 08:24:05.746572551 +0000 UTC m=+301.632243105" Jul 21 08:24:08.681797 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:08.681771 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6688d859f-s89c8" Jul 21 08:24:08.773140 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:08.773110 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:24:08.773480 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:08.773440 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" podUID="338368da-2b90-4ac5-b51a-6a4a47002e6a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 21 08:24:14.669350 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:14.669304 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:24:18.773673 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:18.773625 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" podUID="338368da-2b90-4ac5-b51a-6a4a47002e6a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 21 08:24:28.773871 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:28.773835 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" podUID="338368da-2b90-4ac5-b51a-6a4a47002e6a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 21 08:24:28.774260 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:28.773902 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" Jul 21 08:24:28.774400 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:28.774369 2553 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"63d1ec9e0d1c6b64fefc7be17d036e29687ca5255c2c285f0b78984970f50225"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 21 08:24:28.774456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:28.774421 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" podUID="338368da-2b90-4ac5-b51a-6a4a47002e6a" containerName="service-proxy" containerID="cri-o://63d1ec9e0d1c6b64fefc7be17d036e29687ca5255c2c285f0b78984970f50225" gracePeriod=30 Jul 21 08:24:29.773993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:29.773958 2553 generic.go:356] "Generic (PLEG): container finished" podID="338368da-2b90-4ac5-b51a-6a4a47002e6a" containerID="63d1ec9e0d1c6b64fefc7be17d036e29687ca5255c2c285f0b78984970f50225" exitCode=2 Jul 21 08:24:29.774393 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:29.774037 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" event={"ID":"338368da-2b90-4ac5-b51a-6a4a47002e6a","Type":"ContainerDied","Data":"63d1ec9e0d1c6b64fefc7be17d036e29687ca5255c2c285f0b78984970f50225"} Jul 21 08:24:29.774393 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:29.774097 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cbc4d498-4j8zs" event={"ID":"338368da-2b90-4ac5-b51a-6a4a47002e6a","Type":"ContainerStarted","Data":"16d20e6f6962e62e81d1d097352a47510de4ae9f80d4fb89505ecaef852515a2"} Jul 21 08:24:58.773169 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:58.773086 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:24:58.793429 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:58.793404 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:24:58.897184 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:24:58.897155 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:16.786267 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.786234 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:25:16.786707 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.786575 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="prometheus" containerID="cri-o://1aad9c40b06b46287defea1fc8b29fc2fba45199c4fcfd71200841636122aa1d" gracePeriod=600 Jul 21 08:25:16.786707 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.786585 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy" containerID="cri-o://353b68750f036b77cd569bdc5a3d080a2ac376aab5e288a7b00a335896cff07d" gracePeriod=600 Jul 21 08:25:16.786707 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.786609 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6dce055f2ba4277dd85b32ab5ca86d9fb075fed61d0cedbc83cc9b2995df3cb4" gracePeriod=600 Jul 21 08:25:16.786707 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.786646 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-web" containerID="cri-o://3076447cdbc9bd5e7d26d8d75ad50bcfbd008fadef51ff135f8361dde3c69153" gracePeriod=600 Jul 21 08:25:16.786911 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.786646 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="thanos-sidecar" containerID="cri-o://6fc1b7cd3a8fb0967f2ae17be363f94e7b1568a8575bd09902e421e063cf360a" gracePeriod=600 Jul 21 08:25:16.786911 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.786648 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="config-reloader" containerID="cri-o://dd377287baa50eaea60f7fceb9a3902fa5d1fd9f190f44770aaf89766b818092" gracePeriod=600 Jul 21 08:25:16.945768 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945734 2553 generic.go:356] "Generic (PLEG): container finished" podID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerID="6dce055f2ba4277dd85b32ab5ca86d9fb075fed61d0cedbc83cc9b2995df3cb4" exitCode=0 Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945777 2553 generic.go:356] "Generic (PLEG): container finished" podID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerID="353b68750f036b77cd569bdc5a3d080a2ac376aab5e288a7b00a335896cff07d" exitCode=0 Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945789 2553 generic.go:356] "Generic (PLEG): container finished" podID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerID="3076447cdbc9bd5e7d26d8d75ad50bcfbd008fadef51ff135f8361dde3c69153" exitCode=0 Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945798 2553 generic.go:356] "Generic (PLEG): container finished" podID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerID="6fc1b7cd3a8fb0967f2ae17be363f94e7b1568a8575bd09902e421e063cf360a" exitCode=0 Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945806 2553 generic.go:356] "Generic (PLEG): container finished" podID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerID="dd377287baa50eaea60f7fceb9a3902fa5d1fd9f190f44770aaf89766b818092" exitCode=0 Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945804 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"6dce055f2ba4277dd85b32ab5ca86d9fb075fed61d0cedbc83cc9b2995df3cb4"} Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945870 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"353b68750f036b77cd569bdc5a3d080a2ac376aab5e288a7b00a335896cff07d"} Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945814 2553 generic.go:356] "Generic (PLEG): container finished" podID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerID="1aad9c40b06b46287defea1fc8b29fc2fba45199c4fcfd71200841636122aa1d" exitCode=0 Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945898 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"3076447cdbc9bd5e7d26d8d75ad50bcfbd008fadef51ff135f8361dde3c69153"} Jul 21 08:25:16.945934 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945928 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"6fc1b7cd3a8fb0967f2ae17be363f94e7b1568a8575bd09902e421e063cf360a"} Jul 21 08:25:16.946256 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945953 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"dd377287baa50eaea60f7fceb9a3902fa5d1fd9f190f44770aaf89766b818092"} Jul 21 08:25:16.946256 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:16.945975 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"1aad9c40b06b46287defea1fc8b29fc2fba45199c4fcfd71200841636122aa1d"} Jul 21 08:25:17.023984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.023963 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:17.209870 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.209840 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-trusted-ca-bundle\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210070 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.209885 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-metrics-client-ca\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210070 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.209911 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-2\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210070 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.209942 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-serving-certs-ca-bundle\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210070 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.209985 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-db\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210070 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210008 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-metrics-client-certs\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210070 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210036 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-kubelet-serving-ca-bundle\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210070 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210059 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210131 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-grpc-tls\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210174 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-kube-rbac-proxy\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210201 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2754m\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-kube-api-access-2754m\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210231 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-config-out\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210255 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-web-config\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210279 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210298 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-0\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210354 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210377 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-config\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210405 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-1\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210450 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-thanos-prometheus-http-client-file\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.210458 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210353 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210475 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-tls\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210489 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210507 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-tls-assets\") pod \"972e65ff-3913-4f02-9195-04bf617ef9a8\" (UID: \"972e65ff-3913-4f02-9195-04bf617ef9a8\") " Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210508 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210606 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210760 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210779 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210795 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-metrics-client-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210810 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.211024 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.210823 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.211693 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.211167 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:17.211860 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.211810 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:25:17.212483 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.212458 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:25:17.213482 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.213287 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.213482 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.213439 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-config-out" (OuterVolumeSpecName: "config-out") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:25:17.213631 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.213508 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.214082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.214028 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-kube-api-access-2754m" (OuterVolumeSpecName: "kube-api-access-2754m") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "kube-api-access-2754m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:25:17.214285 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.214259 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.214878 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.214851 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.215378 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.215352 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.215710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.215690 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:25:17.215852 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.215833 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.215926 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.215857 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-config" (OuterVolumeSpecName: "config") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.216086 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.216064 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.223566 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.223544 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-web-config" (OuterVolumeSpecName: "web-config") pod "972e65ff-3913-4f02-9195-04bf617ef9a8" (UID: "972e65ff-3913-4f02-9195-04bf617ef9a8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:25:17.311545 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311505 2553 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-kube-rbac-proxy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311545 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311537 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2754m\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-kube-api-access-2754m\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311545 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311547 2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-config-out\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311545 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311556 2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-web-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311564 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311573 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311581 2553 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-config\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311590 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311598 2553 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311608 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311616 2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/972e65ff-3913-4f02-9195-04bf617ef9a8-tls-assets\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311624 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/972e65ff-3913-4f02-9195-04bf617ef9a8-prometheus-k8s-db\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311632 2553 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-metrics-client-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311640 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.311796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.311649 2553 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/972e65ff-3913-4f02-9195-04bf617ef9a8-secret-grpc-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:25:17.952627 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.952598 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"972e65ff-3913-4f02-9195-04bf617ef9a8","Type":"ContainerDied","Data":"b45a8b184dae106087cd13fd07cbdb252838c61c036889d8b5179dcfdb035758"} Jul 21 08:25:17.952627 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.952627 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:17.953077 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.952668 2553 scope.go:117] "RemoveContainer" containerID="6dce055f2ba4277dd85b32ab5ca86d9fb075fed61d0cedbc83cc9b2995df3cb4" Jul 21 08:25:17.960700 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.960682 2553 scope.go:117] "RemoveContainer" containerID="353b68750f036b77cd569bdc5a3d080a2ac376aab5e288a7b00a335896cff07d" Jul 21 08:25:17.967697 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.967680 2553 scope.go:117] "RemoveContainer" containerID="3076447cdbc9bd5e7d26d8d75ad50bcfbd008fadef51ff135f8361dde3c69153" Jul 21 08:25:17.974938 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.974876 2553 scope.go:117] "RemoveContainer" containerID="6fc1b7cd3a8fb0967f2ae17be363f94e7b1568a8575bd09902e421e063cf360a" Jul 21 08:25:17.975802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.975716 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:25:17.979921 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.979902 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:25:17.981885 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.981870 2553 scope.go:117] "RemoveContainer" containerID="dd377287baa50eaea60f7fceb9a3902fa5d1fd9f190f44770aaf89766b818092" Jul 21 08:25:17.988717 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.988687 2553 scope.go:117] "RemoveContainer" containerID="1aad9c40b06b46287defea1fc8b29fc2fba45199c4fcfd71200841636122aa1d" Jul 21 08:25:17.996113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:17.996094 2553 scope.go:117] "RemoveContainer" containerID="e9ff2800e568d1bdc82dd56a317554b185e39bd1d57a70f449fdc9708472d539" Jul 21 08:25:18.003502 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003486 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:25:18.003744 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003731 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="prometheus" Jul 21 08:25:18.003786 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003747 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="prometheus" Jul 21 08:25:18.003786 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003758 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="init-config-reloader" Jul 21 08:25:18.003786 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003763 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="init-config-reloader" Jul 21 08:25:18.003786 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003771 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy" Jul 21 08:25:18.003786 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003777 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy" Jul 21 08:25:18.003786 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003784 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="config-reloader" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003792 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="config-reloader" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003804 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-web" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003809 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-web" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003818 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="thanos-sidecar" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003823 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="thanos-sidecar" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003830 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-thanos" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003835 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-thanos" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003878 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="prometheus" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003885 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="config-reloader" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003892 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-web" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003897 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="thanos-sidecar" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003903 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy" Jul 21 08:25:18.003949 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.003908 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" containerName="kube-rbac-proxy-thanos" Jul 21 08:25:18.008656 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.008639 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.010833 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.010814 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 08:25:18.010989 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.010971 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-73b63larn9eoe\"" Jul 21 08:25:18.011229 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011214 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 08:25:18.011321 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011290 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 08:25:18.011379 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011361 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 08:25:18.011426 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011413 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 08:25:18.011591 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011561 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d525j\"" Jul 21 08:25:18.011591 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011568 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 08:25:18.011735 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011567 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 08:25:18.011735 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011702 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 08:25:18.011888 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011873 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 08:25:18.011952 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011910 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 08:25:18.012001 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.011966 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 08:25:18.012062 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.012045 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 08:25:18.012107 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.012080 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 08:25:18.014175 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.014154 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 08:25:18.015368 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015348 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-config\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015457 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015385 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015457 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015412 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9891a4e3-56c2-450c-b47d-611674391b2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015457 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015433 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015470 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015486 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015514 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015544 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015572 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015637 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9891a4e3-56c2-450c-b47d-611674391b2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015662 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015736 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015765 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015786 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015812 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015838 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.015866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015861 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.016489 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015887 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.016489 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.015927 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxz27\" (UniqueName: \"kubernetes.io/projected/9891a4e3-56c2-450c-b47d-611674391b2f-kube-api-access-rxz27\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.017183 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.017163 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 08:25:18.021804 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.021785 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:25:18.116758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116660 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.116758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116694 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.116758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116711 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.116758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116726 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.116758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116743 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.116758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116759 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9891a4e3-56c2-450c-b47d-611674391b2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116774 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116805 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116840 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116866 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116888 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116908 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116929 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116949 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.116971 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.117006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxz27\" (UniqueName: \"kubernetes.io/projected/9891a4e3-56c2-450c-b47d-611674391b2f-kube-api-access-rxz27\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.117039 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-config\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.117078 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.117105 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9891a4e3-56c2-450c-b47d-611674391b2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.117168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.117130 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.118018 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.117514 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.118018 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.117749 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.118263 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.118237 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.118646 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.118619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.118953 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.118925 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.119792 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.119591 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.119962 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.119936 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.120892 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.120478 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.120892 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.120801 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.120892 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.120857 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.121076 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.120960 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.121437 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.121391 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.121963 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.121918 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.122504 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.122480 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-config\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.122798 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.122770 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9891a4e3-56c2-450c-b47d-611674391b2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.122984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.122965 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.123148 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.123130 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9891a4e3-56c2-450c-b47d-611674391b2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.123379 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.123362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9891a4e3-56c2-450c-b47d-611674391b2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.123450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.123436 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9891a4e3-56c2-450c-b47d-611674391b2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.126218 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.126202 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxz27\" (UniqueName: \"kubernetes.io/projected/9891a4e3-56c2-450c-b47d-611674391b2f-kube-api-access-rxz27\") pod \"prometheus-k8s-0\" (UID: \"9891a4e3-56c2-450c-b47d-611674391b2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.318755 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.318705 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:25:18.452419 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.452387 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 08:25:18.455837 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:25:18.455811 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9891a4e3_56c2_450c_b47d_611674391b2f.slice/crio-39a5de0de0905a6ef9907aee3bcc154e5b16fe638d59fb9b2dac4f4770cce5b2 WatchSource:0}: Error finding container 39a5de0de0905a6ef9907aee3bcc154e5b16fe638d59fb9b2dac4f4770cce5b2: Status 404 returned error can't find the container with id 39a5de0de0905a6ef9907aee3bcc154e5b16fe638d59fb9b2dac4f4770cce5b2 Jul 21 08:25:18.621993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.621907 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:25:18.624295 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.624275 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bcdb240-7d2d-45c8-b455-e65e7c28a22a-metrics-certs\") pod \"network-metrics-daemon-9m59z\" (UID: \"0bcdb240-7d2d-45c8-b455-e65e7c28a22a\") " pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:25:18.675850 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.675821 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972e65ff-3913-4f02-9195-04bf617ef9a8" path="/var/lib/kubelet/pods/972e65ff-3913-4f02-9195-04bf617ef9a8/volumes" Jul 21 08:25:18.871858 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.871828 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bh656\"" Jul 21 08:25:18.880567 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.880511 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9m59z" Jul 21 08:25:18.959413 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.959382 2553 generic.go:356] "Generic (PLEG): container finished" podID="9891a4e3-56c2-450c-b47d-611674391b2f" containerID="3c87a5bafbdddc9afcf6eca1bc99d8580ceaaf245e4a69d3180d20dfb223a202" exitCode=0 Jul 21 08:25:18.960326 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.959429 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerDied","Data":"3c87a5bafbdddc9afcf6eca1bc99d8580ceaaf245e4a69d3180d20dfb223a202"} Jul 21 08:25:18.960326 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:18.959502 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerStarted","Data":"39a5de0de0905a6ef9907aee3bcc154e5b16fe638d59fb9b2dac4f4770cce5b2"} Jul 21 08:25:19.010965 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.010936 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9m59z"] Jul 21 08:25:19.013444 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:25:19.013422 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcdb240_7d2d_45c8_b455_e65e7c28a22a.slice/crio-bab2cea0f52dc30f52c350db6be582dbc2d9f18ea5cd7f0985b50f0b3f5a3dea WatchSource:0}: Error finding container bab2cea0f52dc30f52c350db6be582dbc2d9f18ea5cd7f0985b50f0b3f5a3dea: Status 404 returned error can't find the container with id bab2cea0f52dc30f52c350db6be582dbc2d9f18ea5cd7f0985b50f0b3f5a3dea Jul 21 08:25:19.015538 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.015522 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:25:19.966029 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.965995 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9m59z" event={"ID":"0bcdb240-7d2d-45c8-b455-e65e7c28a22a","Type":"ContainerStarted","Data":"bab2cea0f52dc30f52c350db6be582dbc2d9f18ea5cd7f0985b50f0b3f5a3dea"} Jul 21 08:25:19.969888 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.969858 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerStarted","Data":"a2b6e94545c90762eb7684d1281ffb683fa4097456fc7621d8ce0d5708757bee"} Jul 21 08:25:19.969995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.969938 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerStarted","Data":"51aba0cd811f5a75abe3af67fa0bde752f7407d6cc25073bb065c7922b9b04e0"} Jul 21 08:25:19.969995 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.969983 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerStarted","Data":"e443b85185c86f1dad140b818e3d592a67bc9336f48a31b7a2003bca7f3e933c"} Jul 21 08:25:19.970065 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.970024 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerStarted","Data":"3f84282d7db81b71cf352e55057169f69aae4d4f0bd313842a0993f04731224f"} Jul 21 08:25:19.970098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.970066 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerStarted","Data":"dd31cc34da887bfe9b2dbf7ed92009ce11b29d2257a3e71a4fdccde7e04cd111"} Jul 21 08:25:19.970130 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.970108 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9891a4e3-56c2-450c-b47d-611674391b2f","Type":"ContainerStarted","Data":"44765c92ba73bc367fca43ccaceb37ef3a9ac0073c76eafeec2b77c7e788bfb7"} Jul 21 08:25:19.996142 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:19.996092 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9773353 podStartE2EDuration="2.996075381s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.740081ms" isStatefulPod=false podCreationTimestamp="2026-07-21 08:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:25:19.995758825 +0000 UTC m=+375.881429374" watchObservedRunningTime="2026-07-21 08:25:19.996075381 +0000 UTC m=+375.881745930" Jul 21 08:25:20.975198 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:20.975156 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9m59z" event={"ID":"0bcdb240-7d2d-45c8-b455-e65e7c28a22a","Type":"ContainerStarted","Data":"37e2bfb945fda6bbeab088856ec82966bc47f673ae63b8b3b6fbd92ca6552fd6"} Jul 21 08:25:20.975609 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:20.975212 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9m59z" event={"ID":"0bcdb240-7d2d-45c8-b455-e65e7c28a22a","Type":"ContainerStarted","Data":"1d22227fb94fa220e4dc3e16c696f2aaea50a90a40be09b4b7859e14e3d8d2e4"} Jul 21 08:25:20.990364 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:20.990284 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9m59z" podStartSLOduration=375.059594896 podStartE2EDuration="6m15.990267805s" totalImagesPullingTime="930.672909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:25:20.989337375 +0000 UTC m=+376.875007941" watchObservedRunningTime="2026-07-21 08:25:20.990267805 +0000 UTC m=+376.875938386" Jul 21 08:25:23.319254 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:25:23.319216 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:26:18.319806 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:26:18.319768 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:26:18.335802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:26:18.335777 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:26:19.200173 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:26:19.200140 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 08:28:54.724927 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.724892 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb"] Jul 21 08:28:54.728281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.728263 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.732807 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.732787 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 08:28:54.732897 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.732830 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 08:28:54.732897 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.732787 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 08:28:54.733207 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.733193 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-568nx\"" Jul 21 08:28:54.734939 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.734917 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 08:28:54.745409 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.745381 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb"] Jul 21 08:28:54.798676 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.798641 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ca8fbc0-8b40-461e-997d-8f03052c473e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.798870 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.798687 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca8fbc0-8b40-461e-997d-8f03052c473e-tmp\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.798870 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.798710 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfcgm\" (UniqueName: \"kubernetes.io/projected/8ca8fbc0-8b40-461e-997d-8f03052c473e-kube-api-access-dfcgm\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.798870 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.798777 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ca8fbc0-8b40-461e-997d-8f03052c473e-manifests\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.798870 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.798804 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ca8fbc0-8b40-461e-997d-8f03052c473e-webhook-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.899847 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.899805 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ca8fbc0-8b40-461e-997d-8f03052c473e-manifests\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.900058 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.899862 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ca8fbc0-8b40-461e-997d-8f03052c473e-webhook-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.900058 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.899918 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ca8fbc0-8b40-461e-997d-8f03052c473e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.900058 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.899949 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca8fbc0-8b40-461e-997d-8f03052c473e-tmp\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.900058 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.899974 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfcgm\" (UniqueName: \"kubernetes.io/projected/8ca8fbc0-8b40-461e-997d-8f03052c473e-kube-api-access-dfcgm\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.900292 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.900264 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ca8fbc0-8b40-461e-997d-8f03052c473e-manifests\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.902383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.902357 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca8fbc0-8b40-461e-997d-8f03052c473e-tmp\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.902643 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.902626 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ca8fbc0-8b40-461e-997d-8f03052c473e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.902696 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.902677 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ca8fbc0-8b40-461e-997d-8f03052c473e-webhook-cert\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:54.907611 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:54.907593 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfcgm\" (UniqueName: \"kubernetes.io/projected/8ca8fbc0-8b40-461e-997d-8f03052c473e-kube-api-access-dfcgm\") pod \"opendatahub-operator-controller-manager-787895fdf5-r45tb\" (UID: \"8ca8fbc0-8b40-461e-997d-8f03052c473e\") " pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:55.038111 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:55.038029 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:28:55.169823 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:55.169788 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb"] Jul 21 08:28:55.173728 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:28:55.173697 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca8fbc0_8b40_461e_997d_8f03052c473e.slice/crio-26a9589a887673aea911ad25b713dde3fe9cf98bb575a4b299c63fa2c572f8ce WatchSource:0}: Error finding container 26a9589a887673aea911ad25b713dde3fe9cf98bb575a4b299c63fa2c572f8ce: Status 404 returned error can't find the container with id 26a9589a887673aea911ad25b713dde3fe9cf98bb575a4b299c63fa2c572f8ce Jul 21 08:28:55.742172 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:55.742141 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" event={"ID":"8ca8fbc0-8b40-461e-997d-8f03052c473e","Type":"ContainerStarted","Data":"26a9589a887673aea911ad25b713dde3fe9cf98bb575a4b299c63fa2c572f8ce"} Jul 21 08:28:58.758754 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:58.758661 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" event={"ID":"8ca8fbc0-8b40-461e-997d-8f03052c473e","Type":"ContainerStarted","Data":"89096ce38d31b60aefbf2e78908c186d3dee519e86f22329a7ab1883c08d0f7b"} Jul 21 08:28:59.763984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:59.763948 2553 generic.go:356] "Generic (PLEG): container finished" podID="8ca8fbc0-8b40-461e-997d-8f03052c473e" containerID="89096ce38d31b60aefbf2e78908c186d3dee519e86f22329a7ab1883c08d0f7b" exitCode=0 Jul 21 08:28:59.764428 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:28:59.764032 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" event={"ID":"8ca8fbc0-8b40-461e-997d-8f03052c473e","Type":"ContainerDied","Data":"89096ce38d31b60aefbf2e78908c186d3dee519e86f22329a7ab1883c08d0f7b"} Jul 21 08:29:00.769551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:00.769512 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" event={"ID":"8ca8fbc0-8b40-461e-997d-8f03052c473e","Type":"ContainerStarted","Data":"60c4a8d8e9b831e36347b6a93a9d2e6b1718c6e0b716d7199c13a55379af7956"} Jul 21 08:29:00.769973 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:00.769734 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:29:00.790812 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:00.790759 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" podStartSLOduration=1.944255436 podStartE2EDuration="6.790742715s" totalImagesPullingTime="3.784936008s" totalInitContainerRuntime="1.061551271s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:28:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:00.788795655 +0000 UTC m=+596.674466203" watchObservedRunningTime="2026-07-21 08:29:00.790742715 +0000 UTC m=+596.676413263" Jul 21 08:29:04.939026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:04.938907 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:29:04.939026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:04.938916 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:29:11.775699 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:11.775673 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-787895fdf5-r45tb" Jul 21 08:29:14.283295 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.283263 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz"] Jul 21 08:29:14.291004 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.290983 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.293190 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.293166 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 08:29:14.293435 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.293419 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 08:29:14.293526 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.293461 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sv5ns\"" Jul 21 08:29:14.294010 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.293994 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 08:29:14.294090 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.294073 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 08:29:14.297166 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.297144 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz"] Jul 21 08:29:14.353621 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.353584 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxrw\" (UniqueName: \"kubernetes.io/projected/e23b9e72-2997-429f-8c4c-63789ddbf0c5-kube-api-access-kkxrw\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.353621 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.353625 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.353850 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.353644 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tmp\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.454838 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.454798 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.454838 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.454842 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tmp\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.455087 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.454932 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxrw\" (UniqueName: \"kubernetes.io/projected/e23b9e72-2997-429f-8c4c-63789ddbf0c5-kube-api-access-kkxrw\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.455087 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:14.454947 2553 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 21 08:29:14.455087 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:14.455018 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tls-certs podName:e23b9e72-2997-429f-8c4c-63789ddbf0c5 nodeName:}" failed. No retries permitted until 2026-07-21 08:29:14.955001819 +0000 UTC m=+610.840672347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tls-certs") pod "kube-auth-proxy-7d57bc4cd9-cwctz" (UID: "e23b9e72-2997-429f-8c4c-63789ddbf0c5") : secret "kube-auth-proxy-tls" not found Jul 21 08:29:14.457208 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.457183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tmp\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.470061 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.470033 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxrw\" (UniqueName: \"kubernetes.io/projected/e23b9e72-2997-429f-8c4c-63789ddbf0c5-kube-api-access-kkxrw\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.521733 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.521703 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h7s4c"] Jul 21 08:29:14.523896 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.523879 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:14.527321 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.527295 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-l8k78\"" Jul 21 08:29:14.529829 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.529809 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 08:29:14.543919 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.543852 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h7s4c"] Jul 21 08:29:14.656078 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.656039 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-cert\") pod \"odh-model-controller-7998c67dc4-h7s4c\" (UID: \"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e\") " pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:14.656258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.656085 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qb7g\" (UniqueName: \"kubernetes.io/projected/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-kube-api-access-9qb7g\") pod \"odh-model-controller-7998c67dc4-h7s4c\" (UID: \"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e\") " pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:14.756531 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.756491 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-cert\") pod \"odh-model-controller-7998c67dc4-h7s4c\" (UID: \"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e\") " pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:14.756730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.756539 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qb7g\" (UniqueName: \"kubernetes.io/projected/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-kube-api-access-9qb7g\") pod \"odh-model-controller-7998c67dc4-h7s4c\" (UID: \"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e\") " pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:14.756730 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:14.756659 2553 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 08:29:14.756842 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:14.756756 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-cert podName:4ed5bb95-54c6-4878-97c9-aa8ea6cc946e nodeName:}" failed. No retries permitted until 2026-07-21 08:29:15.256733104 +0000 UTC m=+611.142403630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-cert") pod "odh-model-controller-7998c67dc4-h7s4c" (UID: "4ed5bb95-54c6-4878-97c9-aa8ea6cc946e") : secret "odh-model-controller-webhook-cert" not found Jul 21 08:29:14.768535 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.768492 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qb7g\" (UniqueName: \"kubernetes.io/projected/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-kube-api-access-9qb7g\") pod \"odh-model-controller-7998c67dc4-h7s4c\" (UID: \"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e\") " pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:14.957528 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.957491 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:14.960136 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:14.960099 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23b9e72-2997-429f-8c4c-63789ddbf0c5-tls-certs\") pod \"kube-auth-proxy-7d57bc4cd9-cwctz\" (UID: \"e23b9e72-2997-429f-8c4c-63789ddbf0c5\") " pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:15.201853 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.201819 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" Jul 21 08:29:15.255005 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.254971 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zbfhp"] Jul 21 08:29:15.257712 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.257690 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:15.260125 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.260057 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-cert\") pod \"odh-model-controller-7998c67dc4-h7s4c\" (UID: \"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e\") " pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:15.260829 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.260571 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-vztd5\"" Jul 21 08:29:15.260829 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.260602 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 08:29:15.262795 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.262752 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ed5bb95-54c6-4878-97c9-aa8ea6cc946e-cert\") pod \"odh-model-controller-7998c67dc4-h7s4c\" (UID: \"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e\") " pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:15.271186 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.271161 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zbfhp"] Jul 21 08:29:15.337723 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.337686 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz"] Jul 21 08:29:15.340562 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:29:15.340530 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23b9e72_2997_429f_8c4c_63789ddbf0c5.slice/crio-7af181b3c38f0510578939e101c3ea93bbc5eee5ea03c3cdff6cf987109c645f WatchSource:0}: Error finding container 7af181b3c38f0510578939e101c3ea93bbc5eee5ea03c3cdff6cf987109c645f: Status 404 returned error can't find the container with id 7af181b3c38f0510578939e101c3ea93bbc5eee5ea03c3cdff6cf987109c645f Jul 21 08:29:15.361007 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.360979 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-tls-certs\") pod \"model-serving-api-54d89bd5fc-zbfhp\" (UID: \"4e8dbd59-fb17-42c1-b8ab-b82b75142d11\") " pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:15.361171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.361013 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzss\" (UniqueName: \"kubernetes.io/projected/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-kube-api-access-tfzss\") pod \"model-serving-api-54d89bd5fc-zbfhp\" (UID: \"4e8dbd59-fb17-42c1-b8ab-b82b75142d11\") " pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:15.434272 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.434230 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:15.462102 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.462006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-tls-certs\") pod \"model-serving-api-54d89bd5fc-zbfhp\" (UID: \"4e8dbd59-fb17-42c1-b8ab-b82b75142d11\") " pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:15.462102 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.462044 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzss\" (UniqueName: \"kubernetes.io/projected/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-kube-api-access-tfzss\") pod \"model-serving-api-54d89bd5fc-zbfhp\" (UID: \"4e8dbd59-fb17-42c1-b8ab-b82b75142d11\") " pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:15.462346 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:15.462163 2553 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 21 08:29:15.462346 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:15.462264 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-tls-certs podName:4e8dbd59-fb17-42c1-b8ab-b82b75142d11 nodeName:}" failed. No retries permitted until 2026-07-21 08:29:15.962232961 +0000 UTC m=+611.847903488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-tls-certs") pod "model-serving-api-54d89bd5fc-zbfhp" (UID: "4e8dbd59-fb17-42c1-b8ab-b82b75142d11") : secret "model-serving-api-tls" not found Jul 21 08:29:15.470899 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.470875 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzss\" (UniqueName: \"kubernetes.io/projected/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-kube-api-access-tfzss\") pod \"model-serving-api-54d89bd5fc-zbfhp\" (UID: \"4e8dbd59-fb17-42c1-b8ab-b82b75142d11\") " pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:15.562941 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.562817 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h7s4c"] Jul 21 08:29:15.565593 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:29:15.565566 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed5bb95_54c6_4878_97c9_aa8ea6cc946e.slice/crio-4750b06fdb68af9e3536f6c604e25492c648512a5103bf3f883a112d66156c3b WatchSource:0}: Error finding container 4750b06fdb68af9e3536f6c604e25492c648512a5103bf3f883a112d66156c3b: Status 404 returned error can't find the container with id 4750b06fdb68af9e3536f6c604e25492c648512a5103bf3f883a112d66156c3b Jul 21 08:29:15.604972 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.604937 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx"] Jul 21 08:29:15.607876 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.607857 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.610538 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.610506 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 08:29:15.610538 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.610521 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q8qzp\"" Jul 21 08:29:15.610712 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.610521 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 08:29:15.610712 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.610514 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 08:29:15.610712 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.610542 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 08:29:15.618886 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.618863 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx"] Jul 21 08:29:15.764689 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.764604 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.764689 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.764643 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-cacerts\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.764689 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.764671 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.764916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.764744 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-local-certs\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.764916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.764782 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nlvl\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-kube-api-access-9nlvl\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.764916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.764851 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.764916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.764869 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-token\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.826088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.826051 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" event={"ID":"e23b9e72-2997-429f-8c4c-63789ddbf0c5","Type":"ContainerStarted","Data":"7af181b3c38f0510578939e101c3ea93bbc5eee5ea03c3cdff6cf987109c645f"} Jul 21 08:29:15.827129 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.827105 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" event={"ID":"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e","Type":"ContainerStarted","Data":"4750b06fdb68af9e3536f6c604e25492c648512a5103bf3f883a112d66156c3b"} Jul 21 08:29:15.865596 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.865557 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.865596 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.865600 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-token\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.865802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.865643 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.865802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.865668 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-cacerts\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.865802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.865693 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.865802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.865720 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-local-certs\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.865802 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.865758 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nlvl\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-kube-api-access-9nlvl\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.866415 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.866375 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.868194 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.868172 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-local-certs\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.868295 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.868226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.868576 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.868558 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.868613 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.868577 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-cacerts\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.879052 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.879026 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nlvl\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-kube-api-access-9nlvl\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.879872 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.879850 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-token\") pod \"istiod-openshift-gateway-7574459656-4rjmx\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.917932 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.917912 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:15.967077 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.967040 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-tls-certs\") pod \"model-serving-api-54d89bd5fc-zbfhp\" (UID: \"4e8dbd59-fb17-42c1-b8ab-b82b75142d11\") " pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:15.970352 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:15.970325 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e8dbd59-fb17-42c1-b8ab-b82b75142d11-tls-certs\") pod \"model-serving-api-54d89bd5fc-zbfhp\" (UID: \"4e8dbd59-fb17-42c1-b8ab-b82b75142d11\") " pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:16.064708 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:16.064564 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx"] Jul 21 08:29:16.072669 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:29:16.072627 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f18f9f_0e70_4d7c_9d56_7639fe453cfd.slice/crio-6cedff5d8c9dc8a0d850e11b2caacdeb6b5c5e467a5a256967a2977785b2aea3 WatchSource:0}: Error finding container 6cedff5d8c9dc8a0d850e11b2caacdeb6b5c5e467a5a256967a2977785b2aea3: Status 404 returned error can't find the container with id 6cedff5d8c9dc8a0d850e11b2caacdeb6b5c5e467a5a256967a2977785b2aea3 Jul 21 08:29:16.173999 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:16.173962 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:16.324689 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:16.324212 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zbfhp"] Jul 21 08:29:16.327698 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:29:16.327664 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8dbd59_fb17_42c1_b8ab_b82b75142d11.slice/crio-747072cc9c5388253fc28da81f6ae84d5a57533598724c2151672844c07002c5 WatchSource:0}: Error finding container 747072cc9c5388253fc28da81f6ae84d5a57533598724c2151672844c07002c5: Status 404 returned error can't find the container with id 747072cc9c5388253fc28da81f6ae84d5a57533598724c2151672844c07002c5 Jul 21 08:29:16.837675 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:16.837536 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" event={"ID":"4e8dbd59-fb17-42c1-b8ab-b82b75142d11","Type":"ContainerStarted","Data":"747072cc9c5388253fc28da81f6ae84d5a57533598724c2151672844c07002c5"} Jul 21 08:29:16.840931 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:16.840845 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" event={"ID":"53f18f9f-0e70-4d7c-9d56-7639fe453cfd","Type":"ContainerStarted","Data":"6cedff5d8c9dc8a0d850e11b2caacdeb6b5c5e467a5a256967a2977785b2aea3"} Jul 21 08:29:20.750890 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.750859 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:29:20.751272 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.750920 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:29:20.865900 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.865363 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" event={"ID":"e23b9e72-2997-429f-8c4c-63789ddbf0c5","Type":"ContainerStarted","Data":"ed83ef867fb5a65e6ab68cc5a5f2d0265a9a0e56551a6b1471058197a796ff7b"} Jul 21 08:29:20.867496 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.867238 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" event={"ID":"4e8dbd59-fb17-42c1-b8ab-b82b75142d11","Type":"ContainerStarted","Data":"357ea9fcda3a4533fb4c8c85ae9248cd8676278ec06faaf6ce3e23bae1400eb7"} Jul 21 08:29:20.867496 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.867295 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:29:20.868878 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.868850 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" event={"ID":"4ed5bb95-54c6-4878-97c9-aa8ea6cc946e","Type":"ContainerStarted","Data":"0ac4c57e109006375d0ca6dcf90d7181a714b8a23e19d468f452ed79a34855d4"} Jul 21 08:29:20.869172 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.868916 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:20.870610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.870574 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" event={"ID":"53f18f9f-0e70-4d7c-9d56-7639fe453cfd","Type":"ContainerStarted","Data":"f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a"} Jul 21 08:29:20.870727 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.870694 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:20.883145 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.883015 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d57bc4cd9-cwctz" podStartSLOduration=1.47697741 podStartE2EDuration="6.882998223s" totalImagesPullingTime="5.406020813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:20.881764761 +0000 UTC m=+616.767435331" watchObservedRunningTime="2026-07-21 08:29:20.882998223 +0000 UTC m=+616.768668773" Jul 21 08:29:20.897435 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.897380 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" podStartSLOduration=1.7074410389999999 podStartE2EDuration="6.897360952s" totalImagesPullingTime="5.189919913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:20.897061749 +0000 UTC m=+616.782732300" watchObservedRunningTime="2026-07-21 08:29:20.897360952 +0000 UTC m=+616.783031500" Jul 21 08:29:20.917379 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.917332 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" podStartSLOduration=1.50080798 podStartE2EDuration="5.917295724s" totalImagesPullingTime="4.416487744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:20.915508134 +0000 UTC m=+616.801178683" watchObservedRunningTime="2026-07-21 08:29:20.917295724 +0000 UTC m=+616.802966272" Jul 21 08:29:20.934852 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:20.934761 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" podStartSLOduration=1.258415647 podStartE2EDuration="5.934739599s" totalImagesPullingTime="4.676323952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:20.933851045 +0000 UTC m=+616.819521620" watchObservedRunningTime="2026-07-21 08:29:20.934739599 +0000 UTC m=+616.820410149" Jul 21 08:29:21.878931 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:21.878897 2553 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-4rjmx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 08:29:21.879327 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:21.878959 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" podUID="53f18f9f-0e70-4d7c-9d56-7639fe453cfd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 08:29:22.406188 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.406150 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f"] Jul 21 08:29:22.409260 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.409241 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.411708 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.411687 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-czrrv\"" Jul 21 08:29:22.422215 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.422192 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f"] Jul 21 08:29:22.532587 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532548 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532587 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532591 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532782 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532782 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532679 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532782 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532782 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532732 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532782 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532764 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qvq\" (UniqueName: \"kubernetes.io/projected/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-kube-api-access-x9qvq\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532937 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.532937 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.532811 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633701 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633653 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633701 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633733 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633759 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633776 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633824 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qvq\" (UniqueName: \"kubernetes.io/projected/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-kube-api-access-x9qvq\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633866 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.633957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.633894 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.634287 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.634105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.634287 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.634137 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.634435 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.634332 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.634493 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.634472 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.634553 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.634536 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.636261 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.636242 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.636353 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.636269 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.646422 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.646380 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.646632 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.646613 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qvq\" (UniqueName: \"kubernetes.io/projected/8e0dd0b4-f3b4-4460-9944-9c5e01b188e1-kube-api-access-x9qvq\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6784n74f\" (UID: \"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.719183 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.719143 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:22.860434 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.860412 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f"] Jul 21 08:29:22.861889 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:29:22.861863 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0dd0b4_f3b4_4460_9944_9c5e01b188e1.slice/crio-e6f9b336bed11c2f267e96e72099960568f4eec5d66192b002b72360be9effc6 WatchSource:0}: Error finding container e6f9b336bed11c2f267e96e72099960568f4eec5d66192b002b72360be9effc6: Status 404 returned error can't find the container with id e6f9b336bed11c2f267e96e72099960568f4eec5d66192b002b72360be9effc6 Jul 21 08:29:22.886125 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:22.886088 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" event={"ID":"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1","Type":"ContainerStarted","Data":"e6f9b336bed11c2f267e96e72099960568f4eec5d66192b002b72360be9effc6"} Jul 21 08:29:24.594665 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.594590 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx"] Jul 21 08:29:24.601199 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.601174 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:24.603588 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.603567 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 08:29:24.603728 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.603568 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7m2gv\"" Jul 21 08:29:24.612194 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.611783 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw"] Jul 21 08:29:24.616144 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.616111 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx"] Jul 21 08:29:24.616288 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.616223 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:24.618654 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.618610 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 08:29:24.618654 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.618641 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-84qhm\"" Jul 21 08:29:24.627456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.627426 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw"] Jul 21 08:29:24.754078 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.754044 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzbd\" (UniqueName: \"kubernetes.io/projected/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-kube-api-access-xlzbd\") pod \"kserve-controller-manager-d6f4b65b8-hmqbx\" (UID: \"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:24.754281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.754095 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmbq\" (UniqueName: \"kubernetes.io/projected/932abbba-9568-4652-b6c3-bc0d9973b14a-kube-api-access-dtmbq\") pod \"llmisvc-controller-manager-777c6bd7d5-jk7dw\" (UID: \"932abbba-9568-4652-b6c3-bc0d9973b14a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:24.754281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.754200 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-cert\") pod \"kserve-controller-manager-d6f4b65b8-hmqbx\" (UID: \"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:24.754281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.754249 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932abbba-9568-4652-b6c3-bc0d9973b14a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jk7dw\" (UID: \"932abbba-9568-4652-b6c3-bc0d9973b14a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:24.855519 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.855424 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzbd\" (UniqueName: \"kubernetes.io/projected/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-kube-api-access-xlzbd\") pod \"kserve-controller-manager-d6f4b65b8-hmqbx\" (UID: \"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:24.855519 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.855488 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmbq\" (UniqueName: \"kubernetes.io/projected/932abbba-9568-4652-b6c3-bc0d9973b14a-kube-api-access-dtmbq\") pod \"llmisvc-controller-manager-777c6bd7d5-jk7dw\" (UID: \"932abbba-9568-4652-b6c3-bc0d9973b14a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:24.855748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.855547 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-cert\") pod \"kserve-controller-manager-d6f4b65b8-hmqbx\" (UID: \"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:24.855748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.855591 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932abbba-9568-4652-b6c3-bc0d9973b14a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jk7dw\" (UID: \"932abbba-9568-4652-b6c3-bc0d9973b14a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:24.855748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:24.855688 2553 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 21 08:29:24.855748 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:24.855695 2553 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 21 08:29:24.855925 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:24.855755 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932abbba-9568-4652-b6c3-bc0d9973b14a-cert podName:932abbba-9568-4652-b6c3-bc0d9973b14a nodeName:}" failed. No retries permitted until 2026-07-21 08:29:25.355733751 +0000 UTC m=+621.241404283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/932abbba-9568-4652-b6c3-bc0d9973b14a-cert") pod "llmisvc-controller-manager-777c6bd7d5-jk7dw" (UID: "932abbba-9568-4652-b6c3-bc0d9973b14a") : secret "llmisvc-webhook-server-cert" not found Jul 21 08:29:24.855925 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:24.855768 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-cert podName:33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a nodeName:}" failed. No retries permitted until 2026-07-21 08:29:25.355762365 +0000 UTC m=+621.241432890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-cert") pod "kserve-controller-manager-d6f4b65b8-hmqbx" (UID: "33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a") : secret "kserve-webhook-server-cert" not found Jul 21 08:29:24.878208 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.878169 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:24.893551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.893513 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmbq\" (UniqueName: \"kubernetes.io/projected/932abbba-9568-4652-b6c3-bc0d9973b14a-kube-api-access-dtmbq\") pod \"llmisvc-controller-manager-777c6bd7d5-jk7dw\" (UID: \"932abbba-9568-4652-b6c3-bc0d9973b14a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:24.894542 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:24.894517 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzbd\" (UniqueName: \"kubernetes.io/projected/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-kube-api-access-xlzbd\") pod \"kserve-controller-manager-d6f4b65b8-hmqbx\" (UID: \"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:25.361517 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.361479 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-cert\") pod \"kserve-controller-manager-d6f4b65b8-hmqbx\" (UID: \"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:25.361719 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.361545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932abbba-9568-4652-b6c3-bc0d9973b14a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jk7dw\" (UID: \"932abbba-9568-4652-b6c3-bc0d9973b14a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:25.364357 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.364291 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a-cert\") pod \"kserve-controller-manager-d6f4b65b8-hmqbx\" (UID: \"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:25.364357 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.364296 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932abbba-9568-4652-b6c3-bc0d9973b14a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jk7dw\" (UID: \"932abbba-9568-4652-b6c3-bc0d9973b14a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:25.516510 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.516470 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:25.533129 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.533096 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:25.584448 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.584411 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:29:25.584560 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.584491 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:29:25.584560 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.584543 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:29:25.672478 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.672452 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx"] Jul 21 08:29:25.675244 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:29:25.675221 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33af3ff0_8b0e_48c1_8468_fbfcde5e7b6a.slice/crio-a3169e620a629fe8360333bd0a4a021c5312703eab489b00eb2b1af89f4b8cee WatchSource:0}: Error finding container a3169e620a629fe8360333bd0a4a021c5312703eab489b00eb2b1af89f4b8cee: Status 404 returned error can't find the container with id a3169e620a629fe8360333bd0a4a021c5312703eab489b00eb2b1af89f4b8cee Jul 21 08:29:25.720362 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.720330 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw"] Jul 21 08:29:25.722827 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:29:25.722800 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod932abbba_9568_4652_b6c3_bc0d9973b14a.slice/crio-d98a1bf4049d13a21258da861e9cc8329fab2eda0d2e415e55131f3b5896753d WatchSource:0}: Error finding container d98a1bf4049d13a21258da861e9cc8329fab2eda0d2e415e55131f3b5896753d: Status 404 returned error can't find the container with id d98a1bf4049d13a21258da861e9cc8329fab2eda0d2e415e55131f3b5896753d Jul 21 08:29:25.902174 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.902086 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" event={"ID":"932abbba-9568-4652-b6c3-bc0d9973b14a","Type":"ContainerStarted","Data":"d98a1bf4049d13a21258da861e9cc8329fab2eda0d2e415e55131f3b5896753d"} Jul 21 08:29:25.903587 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.903559 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" event={"ID":"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a","Type":"ContainerStarted","Data":"a3169e620a629fe8360333bd0a4a021c5312703eab489b00eb2b1af89f4b8cee"} Jul 21 08:29:25.905351 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.905300 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" event={"ID":"8e0dd0b4-f3b4-4460-9944-9c5e01b188e1","Type":"ContainerStarted","Data":"bc3022bc416e978114b9dab01a03f1125d12423b32a4eaa3d9e314a25aca8f18"} Jul 21 08:29:25.951645 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:25.951595 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" podStartSLOduration=1.231347396 podStartE2EDuration="3.951578903s" totalImagesPullingTime="2.720231507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:25.950576157 +0000 UTC m=+621.836246725" watchObservedRunningTime="2026-07-21 08:29:25.951578903 +0000 UTC m=+621.837249450" Jul 21 08:29:26.305172 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.305123 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx"] Jul 21 08:29:26.305927 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.305672 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" podUID="53f18f9f-0e70-4d7c-9d56-7639fe453cfd" containerName="discovery" containerID="cri-o://f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a" gracePeriod=30 Jul 21 08:29:26.554244 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.554221 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:26.672135 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.672102 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-token\") pod \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " Jul 21 08:29:26.672346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.672183 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-cacerts\") pod \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " Jul 21 08:29:26.672346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.672216 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-ca-configmap\") pod \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " Jul 21 08:29:26.672346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.672274 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-kubeconfig\") pod \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " Jul 21 08:29:26.672346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.672330 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-dns-cert\") pod \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " Jul 21 08:29:26.672879 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.672366 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-local-certs\") pod \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " Jul 21 08:29:26.672879 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.672394 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nlvl\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-kube-api-access-9nlvl\") pod \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\" (UID: \"53f18f9f-0e70-4d7c-9d56-7639fe453cfd\") " Jul 21 08:29:26.673639 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.673566 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "53f18f9f-0e70-4d7c-9d56-7639fe453cfd" (UID: "53f18f9f-0e70-4d7c-9d56-7639fe453cfd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:29:26.675213 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.675181 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-token" (OuterVolumeSpecName: "istio-token") pod "53f18f9f-0e70-4d7c-9d56-7639fe453cfd" (UID: "53f18f9f-0e70-4d7c-9d56-7639fe453cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:29:26.675347 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.675251 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-kube-api-access-9nlvl" (OuterVolumeSpecName: "kube-api-access-9nlvl") pod "53f18f9f-0e70-4d7c-9d56-7639fe453cfd" (UID: "53f18f9f-0e70-4d7c-9d56-7639fe453cfd"). InnerVolumeSpecName "kube-api-access-9nlvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:29:26.675465 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.675447 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-cacerts" (OuterVolumeSpecName: "cacerts") pod "53f18f9f-0e70-4d7c-9d56-7639fe453cfd" (UID: "53f18f9f-0e70-4d7c-9d56-7639fe453cfd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:29:26.675713 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.675677 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-local-certs" (OuterVolumeSpecName: "local-certs") pod "53f18f9f-0e70-4d7c-9d56-7639fe453cfd" (UID: "53f18f9f-0e70-4d7c-9d56-7639fe453cfd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:29:26.675804 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.675728 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "53f18f9f-0e70-4d7c-9d56-7639fe453cfd" (UID: "53f18f9f-0e70-4d7c-9d56-7639fe453cfd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:29:26.677012 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.676977 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "53f18f9f-0e70-4d7c-9d56-7639fe453cfd" (UID: "53f18f9f-0e70-4d7c-9d56-7639fe453cfd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:29:26.720240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.720205 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:26.725831 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.725805 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:26.773968 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.773935 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:29:26.773968 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.773963 2553 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-cacerts\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:29:26.773968 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.773972 2553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-ca-configmap\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:29:26.774183 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.773982 2553 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-kubeconfig\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:29:26.774183 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.773991 2553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-istio-csr-dns-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:29:26.774183 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.774001 2553 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-local-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:29:26.774183 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.774009 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nlvl\" (UniqueName: \"kubernetes.io/projected/53f18f9f-0e70-4d7c-9d56-7639fe453cfd-kube-api-access-9nlvl\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:29:26.913377 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.913341 2553 generic.go:356] "Generic (PLEG): container finished" podID="53f18f9f-0e70-4d7c-9d56-7639fe453cfd" containerID="f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a" exitCode=0 Jul 21 08:29:26.913559 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.913389 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" event={"ID":"53f18f9f-0e70-4d7c-9d56-7639fe453cfd","Type":"ContainerDied","Data":"f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a"} Jul 21 08:29:26.913559 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.913415 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" Jul 21 08:29:26.913559 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.913452 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx" event={"ID":"53f18f9f-0e70-4d7c-9d56-7639fe453cfd","Type":"ContainerDied","Data":"6cedff5d8c9dc8a0d850e11b2caacdeb6b5c5e467a5a256967a2977785b2aea3"} Jul 21 08:29:26.913559 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.913482 2553 scope.go:117] "RemoveContainer" containerID="f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a" Jul 21 08:29:26.914104 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.913902 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:26.914944 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.914912 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6784n74f" Jul 21 08:29:26.954862 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.954830 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx"] Jul 21 08:29:26.958175 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:26.958143 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4rjmx"] Jul 21 08:29:27.088137 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:27.088100 2553 scope.go:117] "RemoveContainer" containerID="f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a" Jul 21 08:29:27.088623 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:29:27.088547 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a\": container with ID starting with f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a not found: ID does not exist" containerID="f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a" Jul 21 08:29:27.088623 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:27.088591 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a"} err="failed to get container status \"f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a\": rpc error: code = NotFound desc = could not find container \"f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a\": container with ID starting with f8b909cd25e0295aa64e3c1104222de9efb69504539c38132b7e5df288e4c71a not found: ID does not exist" Jul 21 08:29:28.674706 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:28.674676 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f18f9f-0e70-4d7c-9d56-7639fe453cfd" path="/var/lib/kubelet/pods/53f18f9f-0e70-4d7c-9d56-7639fe453cfd/volumes" Jul 21 08:29:29.932157 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:29.932124 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" event={"ID":"932abbba-9568-4652-b6c3-bc0d9973b14a","Type":"ContainerStarted","Data":"4e64bd2349142cca1e580f496557671f68ccafc535e63b7d47af296f6e76ecce"} Jul 21 08:29:29.932639 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:29.932264 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:29:29.933596 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:29.933569 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" event={"ID":"33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a","Type":"ContainerStarted","Data":"24f2eae5279319a31de73e749e88c4a04886f3e081334eb5a004c9f1c024d755"} Jul 21 08:29:29.933728 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:29.933616 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:29:29.948038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:29.947987 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" podStartSLOduration=2.643540746 podStartE2EDuration="5.94797243s" totalImagesPullingTime="3.304431684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:29.947230348 +0000 UTC m=+625.832900919" watchObservedRunningTime="2026-07-21 08:29:29.94797243 +0000 UTC m=+625.833643030" Jul 21 08:29:29.961976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:29.961928 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" podStartSLOduration=2.613913934 podStartE2EDuration="5.961912395s" totalImagesPullingTime="3.347998461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:29:29.961112905 +0000 UTC m=+625.846783452" watchObservedRunningTime="2026-07-21 08:29:29.961912395 +0000 UTC m=+625.847582921" Jul 21 08:29:31.879182 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:31.879153 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-h7s4c" Jul 21 08:29:31.881095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:29:31.881074 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zbfhp" Jul 21 08:30:00.940285 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:00.940250 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jk7dw" Jul 21 08:30:00.943267 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:00.943240 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hmqbx" Jul 21 08:30:15.340710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.340667 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mn5g"] Jul 21 08:30:15.341166 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.341039 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f18f9f-0e70-4d7c-9d56-7639fe453cfd" containerName="discovery" Jul 21 08:30:15.341166 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.341055 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f18f9f-0e70-4d7c-9d56-7639fe453cfd" containerName="discovery" Jul 21 08:30:15.341166 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.341112 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f18f9f-0e70-4d7c-9d56-7639fe453cfd" containerName="discovery" Jul 21 08:30:15.347193 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.347158 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" Jul 21 08:30:15.350097 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.349944 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-wsl4z\"" Jul 21 08:30:15.350266 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.350202 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 08:30:15.350394 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.350264 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 08:30:15.351222 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.351197 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mn5g"] Jul 21 08:30:15.398394 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.398355 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57xc\" (UniqueName: \"kubernetes.io/projected/566a958c-33f2-4412-85c7-8e10014b828b-kube-api-access-c57xc\") pod \"kuadrant-operator-catalog-7mn5g\" (UID: \"566a958c-33f2-4412-85c7-8e10014b828b\") " pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" Jul 21 08:30:15.499223 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.499183 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57xc\" (UniqueName: \"kubernetes.io/projected/566a958c-33f2-4412-85c7-8e10014b828b-kube-api-access-c57xc\") pod \"kuadrant-operator-catalog-7mn5g\" (UID: \"566a958c-33f2-4412-85c7-8e10014b828b\") " pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" Jul 21 08:30:15.506785 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.506760 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57xc\" (UniqueName: \"kubernetes.io/projected/566a958c-33f2-4412-85c7-8e10014b828b-kube-api-access-c57xc\") pod \"kuadrant-operator-catalog-7mn5g\" (UID: \"566a958c-33f2-4412-85c7-8e10014b828b\") " pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" Jul 21 08:30:15.659477 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.659385 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" Jul 21 08:30:15.714433 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.714400 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mn5g"] Jul 21 08:30:15.788611 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.788508 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mn5g"] Jul 21 08:30:15.791284 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:30:15.791259 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566a958c_33f2_4412_85c7_8e10014b828b.slice/crio-fb37c8173df2085fde7a39698fb2fc7a9d51e00bafb0a5649916c3f6a3648f8a WatchSource:0}: Error finding container fb37c8173df2085fde7a39698fb2fc7a9d51e00bafb0a5649916c3f6a3648f8a: Status 404 returned error can't find the container with id fb37c8173df2085fde7a39698fb2fc7a9d51e00bafb0a5649916c3f6a3648f8a Jul 21 08:30:15.925329 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.925289 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dtc5h"] Jul 21 08:30:15.928420 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.928398 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:15.937377 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:15.937354 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dtc5h"] Jul 21 08:30:16.003616 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:16.003581 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cklh\" (UniqueName: \"kubernetes.io/projected/72a37a62-b58f-4410-b29e-49b88d4dd938-kube-api-access-2cklh\") pod \"kuadrant-operator-catalog-dtc5h\" (UID: \"72a37a62-b58f-4410-b29e-49b88d4dd938\") " pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:16.104629 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:16.104578 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cklh\" (UniqueName: \"kubernetes.io/projected/72a37a62-b58f-4410-b29e-49b88d4dd938-kube-api-access-2cklh\") pod \"kuadrant-operator-catalog-dtc5h\" (UID: \"72a37a62-b58f-4410-b29e-49b88d4dd938\") " pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:16.127886 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:16.127856 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cklh\" (UniqueName: \"kubernetes.io/projected/72a37a62-b58f-4410-b29e-49b88d4dd938-kube-api-access-2cklh\") pod \"kuadrant-operator-catalog-dtc5h\" (UID: \"72a37a62-b58f-4410-b29e-49b88d4dd938\") " pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:16.140492 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:16.140461 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" event={"ID":"566a958c-33f2-4412-85c7-8e10014b828b","Type":"ContainerStarted","Data":"fb37c8173df2085fde7a39698fb2fc7a9d51e00bafb0a5649916c3f6a3648f8a"} Jul 21 08:30:16.237422 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:16.237304 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:16.377893 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:16.377866 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dtc5h"] Jul 21 08:30:16.384380 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:30:16.384344 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a37a62_b58f_4410_b29e_49b88d4dd938.slice/crio-27e1f255c1ba50d8a911c59add1516d3f81e567786d627a4b0b0ee82e3441692 WatchSource:0}: Error finding container 27e1f255c1ba50d8a911c59add1516d3f81e567786d627a4b0b0ee82e3441692: Status 404 returned error can't find the container with id 27e1f255c1ba50d8a911c59add1516d3f81e567786d627a4b0b0ee82e3441692 Jul 21 08:30:17.148367 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:17.148302 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" event={"ID":"72a37a62-b58f-4410-b29e-49b88d4dd938","Type":"ContainerStarted","Data":"27e1f255c1ba50d8a911c59add1516d3f81e567786d627a4b0b0ee82e3441692"} Jul 21 08:30:18.154167 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.154133 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" event={"ID":"566a958c-33f2-4412-85c7-8e10014b828b","Type":"ContainerStarted","Data":"5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f"} Jul 21 08:30:18.154703 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.154171 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" podUID="566a958c-33f2-4412-85c7-8e10014b828b" containerName="registry-server" containerID="cri-o://5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f" gracePeriod=2 Jul 21 08:30:18.155724 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.155700 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" event={"ID":"72a37a62-b58f-4410-b29e-49b88d4dd938","Type":"ContainerStarted","Data":"32d8f0162a5c81cac34be31cf0ab881b77c0ee0d71dd8b8a23c967632565eeb3"} Jul 21 08:30:18.172179 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.171534 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" podStartSLOduration=1.318976862 podStartE2EDuration="3.171513488s" totalImagesPullingTime="1.852536626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:30:18.171268455 +0000 UTC m=+674.056939002" watchObservedRunningTime="2026-07-21 08:30:18.171513488 +0000 UTC m=+674.057184037" Jul 21 08:30:18.188739 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.188670 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" podStartSLOduration=1.9284946459999999 podStartE2EDuration="3.188652327s" totalImagesPullingTime="1.260157681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:30:18.187692157 +0000 UTC m=+674.073362723" watchObservedRunningTime="2026-07-21 08:30:18.188652327 +0000 UTC m=+674.074322874" Jul 21 08:30:18.395132 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.395108 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" Jul 21 08:30:18.422483 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.422457 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c57xc\" (UniqueName: \"kubernetes.io/projected/566a958c-33f2-4412-85c7-8e10014b828b-kube-api-access-c57xc\") pod \"566a958c-33f2-4412-85c7-8e10014b828b\" (UID: \"566a958c-33f2-4412-85c7-8e10014b828b\") " Jul 21 08:30:18.424691 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.424667 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566a958c-33f2-4412-85c7-8e10014b828b-kube-api-access-c57xc" (OuterVolumeSpecName: "kube-api-access-c57xc") pod "566a958c-33f2-4412-85c7-8e10014b828b" (UID: "566a958c-33f2-4412-85c7-8e10014b828b"). InnerVolumeSpecName "kube-api-access-c57xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:30:18.523202 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:18.523165 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57xc\" (UniqueName: \"kubernetes.io/projected/566a958c-33f2-4412-85c7-8e10014b828b-kube-api-access-c57xc\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:30:19.160889 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.160797 2553 generic.go:356] "Generic (PLEG): container finished" podID="566a958c-33f2-4412-85c7-8e10014b828b" containerID="5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f" exitCode=0 Jul 21 08:30:19.160889 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.160838 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" Jul 21 08:30:19.161402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.160882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" event={"ID":"566a958c-33f2-4412-85c7-8e10014b828b","Type":"ContainerDied","Data":"5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f"} Jul 21 08:30:19.161402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.160936 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7mn5g" event={"ID":"566a958c-33f2-4412-85c7-8e10014b828b","Type":"ContainerDied","Data":"fb37c8173df2085fde7a39698fb2fc7a9d51e00bafb0a5649916c3f6a3648f8a"} Jul 21 08:30:19.161402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.160960 2553 scope.go:117] "RemoveContainer" containerID="5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f" Jul 21 08:30:19.172820 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.172800 2553 scope.go:117] "RemoveContainer" containerID="5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f" Jul 21 08:30:19.173140 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:30:19.173121 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f\": container with ID starting with 5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f not found: ID does not exist" containerID="5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f" Jul 21 08:30:19.173192 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.173150 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f"} err="failed to get container status \"5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f\": rpc error: code = NotFound desc = could not find container \"5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f\": container with ID starting with 5a352695d1071aa77ce0d99c00a48affa5b98bf1cffa365e388e1343fa98912f not found: ID does not exist" Jul 21 08:30:19.176456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.176432 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mn5g"] Jul 21 08:30:19.178455 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:19.178433 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7mn5g"] Jul 21 08:30:20.673613 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:20.673578 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566a958c-33f2-4412-85c7-8e10014b828b" path="/var/lib/kubelet/pods/566a958c-33f2-4412-85c7-8e10014b828b/volumes" Jul 21 08:30:26.238243 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:26.238201 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:26.238690 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:26.238275 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:26.261365 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:26.261332 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:27.218098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:27.218068 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dtc5h" Jul 21 08:30:49.131726 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.131691 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g"] Jul 21 08:30:49.132196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.132043 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566a958c-33f2-4412-85c7-8e10014b828b" containerName="registry-server" Jul 21 08:30:49.132196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.132060 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="566a958c-33f2-4412-85c7-8e10014b828b" containerName="registry-server" Jul 21 08:30:49.132196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.132121 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="566a958c-33f2-4412-85c7-8e10014b828b" containerName="registry-server" Jul 21 08:30:49.134553 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.134538 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.136763 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.136742 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c7k9p\"" Jul 21 08:30:49.148298 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.148265 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g"] Jul 21 08:30:49.285291 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.285248 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9818bc46-1d4c-4388-8d73-0294bce1f031-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.285494 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.285404 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vg5\" (UniqueName: \"kubernetes.io/projected/9818bc46-1d4c-4388-8d73-0294bce1f031-kube-api-access-69vg5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.386412 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.386300 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vg5\" (UniqueName: \"kubernetes.io/projected/9818bc46-1d4c-4388-8d73-0294bce1f031-kube-api-access-69vg5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.386412 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.386384 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9818bc46-1d4c-4388-8d73-0294bce1f031-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.386721 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.386706 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9818bc46-1d4c-4388-8d73-0294bce1f031-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.410967 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.410933 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vg5\" (UniqueName: \"kubernetes.io/projected/9818bc46-1d4c-4388-8d73-0294bce1f031-kube-api-access-69vg5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.444909 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.444868 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:49.605413 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.605190 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g"] Jul 21 08:30:49.607957 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:30:49.607927 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9818bc46_1d4c_4388_8d73_0294bce1f031.slice/crio-1f620aa3e622a626c3703e7db83c0fcd754b9e3f90cdd7bcd1c325ccdc9b15c8 WatchSource:0}: Error finding container 1f620aa3e622a626c3703e7db83c0fcd754b9e3f90cdd7bcd1c325ccdc9b15c8: Status 404 returned error can't find the container with id 1f620aa3e622a626c3703e7db83c0fcd754b9e3f90cdd7bcd1c325ccdc9b15c8 Jul 21 08:30:49.610332 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:49.610298 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:30:50.295505 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:50.295469 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" event={"ID":"9818bc46-1d4c-4388-8d73-0294bce1f031","Type":"ContainerStarted","Data":"1f620aa3e622a626c3703e7db83c0fcd754b9e3f90cdd7bcd1c325ccdc9b15c8"} Jul 21 08:30:51.124404 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.124364 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4"] Jul 21 08:30:51.130972 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.130938 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:30:51.134765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.134736 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-h2tx4\"" Jul 21 08:30:51.145412 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.145385 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4"] Jul 21 08:30:51.202178 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.202145 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xsz\" (UniqueName: \"kubernetes.io/projected/eee641a2-e7b1-4974-afb9-eee783c51648-kube-api-access-75xsz\") pod \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" (UID: \"eee641a2-e7b1-4974-afb9-eee783c51648\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:30:51.303358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.303303 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xsz\" (UniqueName: \"kubernetes.io/projected/eee641a2-e7b1-4974-afb9-eee783c51648-kube-api-access-75xsz\") pod \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" (UID: \"eee641a2-e7b1-4974-afb9-eee783c51648\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:30:51.314423 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.314396 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xsz\" (UniqueName: \"kubernetes.io/projected/eee641a2-e7b1-4974-afb9-eee783c51648-kube-api-access-75xsz\") pod \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" (UID: \"eee641a2-e7b1-4974-afb9-eee783c51648\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:30:51.445045 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.445014 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:30:51.597026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:51.597001 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4"] Jul 21 08:30:51.599462 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:30:51.599433 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee641a2_e7b1_4974_afb9_eee783c51648.slice/crio-13fda4e0c9cbd56705dd3316e1a915f255df13361df151b0a2da6f991f838f06 WatchSource:0}: Error finding container 13fda4e0c9cbd56705dd3316e1a915f255df13361df151b0a2da6f991f838f06: Status 404 returned error can't find the container with id 13fda4e0c9cbd56705dd3316e1a915f255df13361df151b0a2da6f991f838f06 Jul 21 08:30:52.305061 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:52.305024 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" event={"ID":"eee641a2-e7b1-4974-afb9-eee783c51648","Type":"ContainerStarted","Data":"13fda4e0c9cbd56705dd3316e1a915f255df13361df151b0a2da6f991f838f06"} Jul 21 08:30:53.632608 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.632570 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2"] Jul 21 08:30:53.638588 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.638561 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" Jul 21 08:30:53.640965 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.640938 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2sf2g\"" Jul 21 08:30:53.645374 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.645341 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2"] Jul 21 08:30:53.726661 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.726616 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhqdv\" (UniqueName: \"kubernetes.io/projected/b34b31f5-3c43-4a36-9dbc-eb34c976c05a-kube-api-access-lhqdv\") pod \"authorino-operator-75cc8d5cb7-gjwt2\" (UID: \"b34b31f5-3c43-4a36-9dbc-eb34c976c05a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" Jul 21 08:30:53.827620 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.827579 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhqdv\" (UniqueName: \"kubernetes.io/projected/b34b31f5-3c43-4a36-9dbc-eb34c976c05a-kube-api-access-lhqdv\") pod \"authorino-operator-75cc8d5cb7-gjwt2\" (UID: \"b34b31f5-3c43-4a36-9dbc-eb34c976c05a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" Jul 21 08:30:53.842428 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.842398 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhqdv\" (UniqueName: \"kubernetes.io/projected/b34b31f5-3c43-4a36-9dbc-eb34c976c05a-kube-api-access-lhqdv\") pod \"authorino-operator-75cc8d5cb7-gjwt2\" (UID: \"b34b31f5-3c43-4a36-9dbc-eb34c976c05a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" Jul 21 08:30:53.952555 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:53.952524 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" Jul 21 08:30:54.092704 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:54.092601 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2"] Jul 21 08:30:54.094939 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:30:54.094912 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34b31f5_3c43_4a36_9dbc_eb34c976c05a.slice/crio-70c3de40670edf527b30edfcc9dcd9b532a77cf8917469b0579f7a7363258ad4 WatchSource:0}: Error finding container 70c3de40670edf527b30edfcc9dcd9b532a77cf8917469b0579f7a7363258ad4: Status 404 returned error can't find the container with id 70c3de40670edf527b30edfcc9dcd9b532a77cf8917469b0579f7a7363258ad4 Jul 21 08:30:54.314258 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:54.314180 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" event={"ID":"b34b31f5-3c43-4a36-9dbc-eb34c976c05a","Type":"ContainerStarted","Data":"70c3de40670edf527b30edfcc9dcd9b532a77cf8917469b0579f7a7363258ad4"} Jul 21 08:30:56.327781 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:56.327704 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" event={"ID":"9818bc46-1d4c-4388-8d73-0294bce1f031","Type":"ContainerStarted","Data":"67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9"} Jul 21 08:30:56.327781 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:56.327783 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:30:56.355450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:56.355388 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" podStartSLOduration=0.995373828 podStartE2EDuration="7.355365914s" totalImagesPullingTime="6.359992086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:30:56.353299448 +0000 UTC m=+712.238969997" watchObservedRunningTime="2026-07-21 08:30:56.355365914 +0000 UTC m=+712.241036462" Jul 21 08:30:58.345095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:58.345025 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" event={"ID":"eee641a2-e7b1-4974-afb9-eee783c51648","Type":"ContainerStarted","Data":"b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab"} Jul 21 08:30:58.345559 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:58.345262 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:30:58.346704 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:58.346671 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" event={"ID":"b34b31f5-3c43-4a36-9dbc-eb34c976c05a","Type":"ContainerStarted","Data":"712ad657cd0f815f64f006d8e6495c876f55786aad613bbd179caba34597d253"} Jul 21 08:30:58.346832 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:58.346811 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" Jul 21 08:30:58.373297 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:58.373243 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" podStartSLOduration=1.074674056 podStartE2EDuration="7.373227082s" totalImagesPullingTime="6.298553026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:30:58.372109511 +0000 UTC m=+714.257780084" watchObservedRunningTime="2026-07-21 08:30:58.373227082 +0000 UTC m=+714.258897630" Jul 21 08:30:58.415370 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:30:58.415298 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" podStartSLOduration=1.612601773 podStartE2EDuration="5.415281184s" totalImagesPullingTime="3.802679411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:30:58.414222231 +0000 UTC m=+714.299892794" watchObservedRunningTime="2026-07-21 08:30:58.415281184 +0000 UTC m=+714.300951732" Jul 21 08:31:07.335482 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:07.335450 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.311138 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.311099 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq"] Jul 21 08:31:09.316240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.316216 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:09.330255 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.330220 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq"] Jul 21 08:31:09.333830 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.333802 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g"] Jul 21 08:31:09.334041 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.334014 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" containerName="manager" containerID="cri-o://67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9" gracePeriod=2 Jul 21 08:31:09.340225 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.340194 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g"] Jul 21 08:31:09.350513 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.350475 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.354824 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.354798 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:09.354964 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.354885 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gjwt2" Jul 21 08:31:09.363522 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.363488 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq"] Jul 21 08:31:09.363844 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.363821 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-pnwxn], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-pnwxn]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" Jul 21 08:31:09.367500 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.367436 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96"] Jul 21 08:31:09.367955 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.367933 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" containerName="manager" Jul 21 08:31:09.367955 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.367956 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" containerName="manager" Jul 21 08:31:09.368127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.368036 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" containerName="manager" Jul 21 08:31:09.371199 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.371175 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq"] Jul 21 08:31:09.371351 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.371272 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.381475 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.381441 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.390205 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.390174 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96"] Jul 21 08:31:09.393799 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.393765 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4"] Jul 21 08:31:09.403383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.403356 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:09.403538 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.403435 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" containerName="manager" containerID="cri-o://b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab" gracePeriod=2 Jul 21 08:31:09.411328 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.410695 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-x62sq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:09.412651 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.412594 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:09.412974 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.412879 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4"] Jul 21 08:31:09.415205 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.415184 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:09.437342 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.437286 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r"] Jul 21 08:31:09.437879 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.437857 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" containerName="manager" Jul 21 08:31:09.437939 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.437883 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" containerName="manager" Jul 21 08:31:09.438006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.437994 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" containerName="manager" Jul 21 08:31:09.441249 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.441224 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" Jul 21 08:31:09.449248 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.449210 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.451502 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.451468 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:09.469003 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.468966 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwldv\" (UniqueName: \"kubernetes.io/projected/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-kube-api-access-lwldv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9gr96\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.469441 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.469416 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9gr96\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.470622 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.470600 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r"] Jul 21 08:31:09.513942 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.513908 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.516128 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.516099 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-x62sq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:09.570529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.570447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwldv\" (UniqueName: \"kubernetes.io/projected/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-kube-api-access-lwldv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9gr96\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.570529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.570491 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8lb\" (UniqueName: \"kubernetes.io/projected/5aabb72b-458c-4113-86ce-d99307b69515-kube-api-access-hs8lb\") pod \"limitador-operator-controller-manager-859d7b78bd-g698r\" (UID: \"5aabb72b-458c-4113-86ce-d99307b69515\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" Jul 21 08:31:09.570529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.570522 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9gr96\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.570891 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.570873 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9gr96\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.584972 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.584941 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwldv\" (UniqueName: \"kubernetes.io/projected/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-kube-api-access-lwldv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9gr96\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.665101 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.665075 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:09.667284 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.667253 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.667875 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.667860 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.669511 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.669487 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-x62sq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:09.671475 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.671444 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:09.671590 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.671547 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8lb\" (UniqueName: \"kubernetes.io/projected/5aabb72b-458c-4113-86ce-d99307b69515-kube-api-access-hs8lb\") pod \"limitador-operator-controller-manager-859d7b78bd-g698r\" (UID: \"5aabb72b-458c-4113-86ce-d99307b69515\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" Jul 21 08:31:09.673373 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.673352 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:09.675190 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.675171 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:09.677076 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:09.677015 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-x62sq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:09.679495 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.679472 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8lb\" (UniqueName: \"kubernetes.io/projected/5aabb72b-458c-4113-86ce-d99307b69515-kube-api-access-hs8lb\") pod \"limitador-operator-controller-manager-859d7b78bd-g698r\" (UID: \"5aabb72b-458c-4113-86ce-d99307b69515\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" Jul 21 08:31:09.706838 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.706811 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:09.772938 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.772906 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9818bc46-1d4c-4388-8d73-0294bce1f031-extensions-socket-volume\") pod \"9818bc46-1d4c-4388-8d73-0294bce1f031\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " Jul 21 08:31:09.773093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.772968 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75xsz\" (UniqueName: \"kubernetes.io/projected/eee641a2-e7b1-4974-afb9-eee783c51648-kube-api-access-75xsz\") pod \"eee641a2-e7b1-4974-afb9-eee783c51648\" (UID: \"eee641a2-e7b1-4974-afb9-eee783c51648\") " Jul 21 08:31:09.773093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.773018 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69vg5\" (UniqueName: \"kubernetes.io/projected/9818bc46-1d4c-4388-8d73-0294bce1f031-kube-api-access-69vg5\") pod \"9818bc46-1d4c-4388-8d73-0294bce1f031\" (UID: \"9818bc46-1d4c-4388-8d73-0294bce1f031\") " Jul 21 08:31:09.773343 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.773301 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9818bc46-1d4c-4388-8d73-0294bce1f031-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9818bc46-1d4c-4388-8d73-0294bce1f031" (UID: "9818bc46-1d4c-4388-8d73-0294bce1f031"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:31:09.776013 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.775983 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9818bc46-1d4c-4388-8d73-0294bce1f031-kube-api-access-69vg5" (OuterVolumeSpecName: "kube-api-access-69vg5") pod "9818bc46-1d4c-4388-8d73-0294bce1f031" (UID: "9818bc46-1d4c-4388-8d73-0294bce1f031"). InnerVolumeSpecName "kube-api-access-69vg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:31:09.776013 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.775988 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee641a2-e7b1-4974-afb9-eee783c51648-kube-api-access-75xsz" (OuterVolumeSpecName: "kube-api-access-75xsz") pod "eee641a2-e7b1-4974-afb9-eee783c51648" (UID: "eee641a2-e7b1-4974-afb9-eee783c51648"). InnerVolumeSpecName "kube-api-access-75xsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:31:09.781218 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.781193 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" Jul 21 08:31:09.862703 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.862672 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96"] Jul 21 08:31:09.864993 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:31:09.864964 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ce7c31_e977_4a01_926e_08ca7c5e5fe6.slice/crio-2cd8367fd89fb71bc200f165801f5f33e3aeb5330856b86601fce774ca8e3dd9 WatchSource:0}: Error finding container 2cd8367fd89fb71bc200f165801f5f33e3aeb5330856b86601fce774ca8e3dd9: Status 404 returned error can't find the container with id 2cd8367fd89fb71bc200f165801f5f33e3aeb5330856b86601fce774ca8e3dd9 Jul 21 08:31:09.874338 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.874296 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9818bc46-1d4c-4388-8d73-0294bce1f031-extensions-socket-volume\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:31:09.874338 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.874339 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75xsz\" (UniqueName: \"kubernetes.io/projected/eee641a2-e7b1-4974-afb9-eee783c51648-kube-api-access-75xsz\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:31:09.874526 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.874348 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69vg5\" (UniqueName: \"kubernetes.io/projected/9818bc46-1d4c-4388-8d73-0294bce1f031-kube-api-access-69vg5\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:31:09.956442 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:09.956415 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r"] Jul 21 08:31:09.958611 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:31:09.958579 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aabb72b_458c_4113_86ce_d99307b69515.slice/crio-b9d37741785a0abf98541f757007c590d5d6296e1f95bb43643a90c68c16780d WatchSource:0}: Error finding container b9d37741785a0abf98541f757007c590d5d6296e1f95bb43643a90c68c16780d: Status 404 returned error can't find the container with id b9d37741785a0abf98541f757007c590d5d6296e1f95bb43643a90c68c16780d Jul 21 08:31:10.409266 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.409230 2553 generic.go:356] "Generic (PLEG): container finished" podID="eee641a2-e7b1-4974-afb9-eee783c51648" containerID="b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab" exitCode=0 Jul 21 08:31:10.409737 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.409279 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:10.409737 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.409357 2553 scope.go:117] "RemoveContainer" containerID="b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab" Jul 21 08:31:10.410831 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.410811 2553 generic.go:356] "Generic (PLEG): container finished" podID="9818bc46-1d4c-4388-8d73-0294bce1f031" containerID="67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9" exitCode=0 Jul 21 08:31:10.410916 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.410857 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:10.411575 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.411554 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:10.412674 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.412646 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" event={"ID":"5aabb72b-458c-4113-86ce-d99307b69515","Type":"ContainerStarted","Data":"23538b332fafa590c8a15ce4a268d05a67ca4819dfcdbafff20ba20a01c4bc58"} Jul 21 08:31:10.412796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.412702 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" event={"ID":"5aabb72b-458c-4113-86ce-d99307b69515","Type":"ContainerStarted","Data":"b9d37741785a0abf98541f757007c590d5d6296e1f95bb43643a90c68c16780d"} Jul 21 08:31:10.412796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.412730 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" Jul 21 08:31:10.413491 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.413458 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-x62sq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:10.414468 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.414432 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" event={"ID":"33ce7c31-e977-4a01-926e-08ca7c5e5fe6","Type":"ContainerStarted","Data":"b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0"} Jul 21 08:31:10.414573 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.414499 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" event={"ID":"33ce7c31-e977-4a01-926e-08ca7c5e5fe6","Type":"ContainerStarted","Data":"2cd8367fd89fb71bc200f165801f5f33e3aeb5330856b86601fce774ca8e3dd9"} Jul 21 08:31:10.414573 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.414454 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:10.414681 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.414671 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:10.415353 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.415326 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:10.417493 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.417463 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-x62sq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:10.419611 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.419589 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:10.420416 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.420401 2553 scope.go:117] "RemoveContainer" containerID="b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab" Jul 21 08:31:10.420673 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.420650 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab\": container with ID starting with b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab not found: ID does not exist" containerID="b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab" Jul 21 08:31:10.420777 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.420681 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab"} err="failed to get container status \"b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab\": rpc error: code = NotFound desc = could not find container \"b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab\": container with ID starting with b029fc34b9a2f461f9ff04314e65705dc0b64d873f04ce2899887f64fdaf2aab not found: ID does not exist" Jul 21 08:31:10.420777 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.420702 2553 scope.go:117] "RemoveContainer" containerID="67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9" Jul 21 08:31:10.429240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.429220 2553 scope.go:117] "RemoveContainer" containerID="67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9" Jul 21 08:31:10.429515 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.429497 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9\": container with ID starting with 67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9 not found: ID does not exist" containerID="67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9" Jul 21 08:31:10.429579 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.429526 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9"} err="failed to get container status \"67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9\": rpc error: code = NotFound desc = could not find container \"67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9\": container with ID starting with 67741f1eb6405817cd05309e7da4bd8b009be6ce47b57bfd41a93b14ada8a9c9 not found: ID does not exist" Jul 21 08:31:10.440615 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.440568 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" podStartSLOduration=1.440553614 podStartE2EDuration="1.440553614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:10.438413905 +0000 UTC m=+726.324084471" watchObservedRunningTime="2026-07-21 08:31:10.440553614 +0000 UTC m=+726.326224162" Jul 21 08:31:10.457665 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.457634 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:10.457794 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.457682 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" podStartSLOduration=1.457667623 podStartE2EDuration="1.457667623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:10.455862712 +0000 UTC m=+726.341533282" watchObservedRunningTime="2026-07-21 08:31:10.457667623 +0000 UTC m=+726.343338172" Jul 21 08:31:10.459457 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.459430 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-fwfl4\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fwfl4" Jul 21 08:31:10.461179 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.461161 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-djk4g\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-djk4g" Jul 21 08:31:10.462774 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:10.462750 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-x62sq\" is forbidden: User \"system:node:ip-10-0-131-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-134.ec2.internal' and this object" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-x62sq" Jul 21 08:31:10.673715 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.673681 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9818bc46-1d4c-4388-8d73-0294bce1f031" path="/var/lib/kubelet/pods/9818bc46-1d4c-4388-8d73-0294bce1f031/volumes" Jul 21 08:31:10.673948 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.673936 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a37a1f9d-25fb-482f-8895-b1d4a8fee24e" path="/var/lib/kubelet/pods/a37a1f9d-25fb-482f-8895-b1d4a8fee24e/volumes" Jul 21 08:31:10.674097 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:10.674085 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee641a2-e7b1-4974-afb9-eee783c51648" path="/var/lib/kubelet/pods/eee641a2-e7b1-4974-afb9-eee783c51648/volumes" Jul 21 08:31:21.423775 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:21.423737 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g698r" Jul 21 08:31:21.424276 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:21.423817 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:26.836655 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:26.836620 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96"] Jul 21 08:31:26.837098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:26.836958 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" podUID="33ce7c31-e977-4a01-926e-08ca7c5e5fe6" containerName="manager" containerID="cri-o://b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0" gracePeriod=10 Jul 21 08:31:27.085344 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.085287 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:27.131112 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.131038 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwldv\" (UniqueName: \"kubernetes.io/projected/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-kube-api-access-lwldv\") pod \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " Jul 21 08:31:27.131112 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.131096 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-extensions-socket-volume\") pod \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\" (UID: \"33ce7c31-e977-4a01-926e-08ca7c5e5fe6\") " Jul 21 08:31:27.131511 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.131476 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "33ce7c31-e977-4a01-926e-08ca7c5e5fe6" (UID: "33ce7c31-e977-4a01-926e-08ca7c5e5fe6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:31:27.133389 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.133357 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-kube-api-access-lwldv" (OuterVolumeSpecName: "kube-api-access-lwldv") pod "33ce7c31-e977-4a01-926e-08ca7c5e5fe6" (UID: "33ce7c31-e977-4a01-926e-08ca7c5e5fe6"). InnerVolumeSpecName "kube-api-access-lwldv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:31:27.232693 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.232644 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwldv\" (UniqueName: \"kubernetes.io/projected/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-kube-api-access-lwldv\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:31:27.232693 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.232687 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/33ce7c31-e977-4a01-926e-08ca7c5e5fe6-extensions-socket-volume\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:31:27.499221 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.499188 2553 generic.go:356] "Generic (PLEG): container finished" podID="33ce7c31-e977-4a01-926e-08ca7c5e5fe6" containerID="b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0" exitCode=0 Jul 21 08:31:27.499409 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.499232 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" Jul 21 08:31:27.499409 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.499266 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" event={"ID":"33ce7c31-e977-4a01-926e-08ca7c5e5fe6","Type":"ContainerDied","Data":"b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0"} Jul 21 08:31:27.499409 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.499341 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96" event={"ID":"33ce7c31-e977-4a01-926e-08ca7c5e5fe6","Type":"ContainerDied","Data":"2cd8367fd89fb71bc200f165801f5f33e3aeb5330856b86601fce774ca8e3dd9"} Jul 21 08:31:27.499409 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.499366 2553 scope.go:117] "RemoveContainer" containerID="b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0" Jul 21 08:31:27.508521 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.508501 2553 scope.go:117] "RemoveContainer" containerID="b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0" Jul 21 08:31:27.508804 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:27.508787 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0\": container with ID starting with b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0 not found: ID does not exist" containerID="b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0" Jul 21 08:31:27.508847 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.508814 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0"} err="failed to get container status \"b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0\": rpc error: code = NotFound desc = could not find container \"b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0\": container with ID starting with b1fb78abd986c0efd5bb536fa8a990b2ec52fa5bf63091807a6c75070de714f0 not found: ID does not exist" Jul 21 08:31:27.531962 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.531929 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96"] Jul 21 08:31:27.538266 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:27.538236 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9gr96"] Jul 21 08:31:28.673992 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:28.673958 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ce7c31-e977-4a01-926e-08ca7c5e5fe6" path="/var/lib/kubelet/pods/33ce7c31-e977-4a01-926e-08ca7c5e5fe6/volumes" Jul 21 08:31:43.358359 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.358324 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5"] Jul 21 08:31:43.358865 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.358846 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ce7c31-e977-4a01-926e-08ca7c5e5fe6" containerName="manager" Jul 21 08:31:43.358939 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.358869 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ce7c31-e977-4a01-926e-08ca7c5e5fe6" containerName="manager" Jul 21 08:31:43.358985 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.358952 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ce7c31-e977-4a01-926e-08ca7c5e5fe6" containerName="manager" Jul 21 08:31:43.362157 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.362133 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.364845 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.364818 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rlbg2\"" Jul 21 08:31:43.381543 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.381513 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5"] Jul 21 08:31:43.474586 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474552 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.474586 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474587 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb90821-604b-40ab-8efb-f82ca783417a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.474841 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474619 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb90821-604b-40ab-8efb-f82ca783417a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.474841 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474705 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb90821-604b-40ab-8efb-f82ca783417a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.474841 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474772 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.474841 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474829 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.475034 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474858 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.475034 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474892 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbdb\" (UniqueName: \"kubernetes.io/projected/9eb90821-604b-40ab-8efb-f82ca783417a-kube-api-access-rtbdb\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.475034 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.474942 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575504 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbdb\" (UniqueName: \"kubernetes.io/projected/9eb90821-604b-40ab-8efb-f82ca783417a-kube-api-access-rtbdb\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575504 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575503 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575740 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575548 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575740 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575567 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb90821-604b-40ab-8efb-f82ca783417a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575740 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575598 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb90821-604b-40ab-8efb-f82ca783417a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575740 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb90821-604b-40ab-8efb-f82ca783417a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575740 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575738 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575789 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.575980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.575940 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.576071 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.576042 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.576112 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.576078 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.576169 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.576150 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.576206 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.576172 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.576659 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.576635 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb90821-604b-40ab-8efb-f82ca783417a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.578483 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.578457 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb90821-604b-40ab-8efb-f82ca783417a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.578595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.578567 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb90821-604b-40ab-8efb-f82ca783417a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.583447 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.583419 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb90821-604b-40ab-8efb-f82ca783417a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.584584 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.584558 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbdb\" (UniqueName: \"kubernetes.io/projected/9eb90821-604b-40ab-8efb-f82ca783417a-kube-api-access-rtbdb\") pod \"maas-default-gateway-openshift-default-5896d4dd49-dk6s5\" (UID: \"9eb90821-604b-40ab-8efb-f82ca783417a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.672856 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.672822 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:43.826452 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.826378 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5"] Jul 21 08:31:43.828967 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:31:43.828940 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb90821_604b_40ab_8efb_f82ca783417a.slice/crio-091975d6dc857edaeeb51bb273b9a65bf536ee33f6aa2617234aabe4cbe81a00 WatchSource:0}: Error finding container 091975d6dc857edaeeb51bb273b9a65bf536ee33f6aa2617234aabe4cbe81a00: Status 404 returned error can't find the container with id 091975d6dc857edaeeb51bb273b9a65bf536ee33f6aa2617234aabe4cbe81a00 Jul 21 08:31:43.831436 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.831405 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:31:43.831513 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.831486 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:31:43.831551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:43.831518 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:31:44.580424 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:44.580387 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" event={"ID":"9eb90821-604b-40ab-8efb-f82ca783417a","Type":"ContainerStarted","Data":"c422f34fbde2e0ace5953dafbc9126f626d91ac2b53aee09b3e968844cff0c65"} Jul 21 08:31:44.580792 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:44.580448 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" event={"ID":"9eb90821-604b-40ab-8efb-f82ca783417a","Type":"ContainerStarted","Data":"091975d6dc857edaeeb51bb273b9a65bf536ee33f6aa2617234aabe4cbe81a00"} Jul 21 08:31:44.601828 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:44.601764 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" podStartSLOduration=1.60174623 podStartE2EDuration="1.60174623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:44.601233284 +0000 UTC m=+760.486903847" watchObservedRunningTime="2026-07-21 08:31:44.60174623 +0000 UTC m=+760.487416778" Jul 21 08:31:44.679301 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:44.679270 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:44.680120 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:44.680098 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:45.584568 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.584532 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:45.585539 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.585515 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:31:45.775326 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.775284 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfmx9"] Jul 21 08:31:45.777871 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.777848 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:45.780254 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.780229 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cc662\"" Jul 21 08:31:45.780424 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.780229 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 08:31:45.787783 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.787759 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfmx9"] Jul 21 08:31:45.883582 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.883490 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfmx9"] Jul 21 08:31:45.896980 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.896952 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwh8k\" (UniqueName: \"kubernetes.io/projected/c90beec2-70e8-45fa-bcb0-055bcb27434d-kube-api-access-xwh8k\") pod \"limitador-limitador-64c94fd64f-cfmx9\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:45.897080 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.896998 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c90beec2-70e8-45fa-bcb0-055bcb27434d-config-file\") pod \"limitador-limitador-64c94fd64f-cfmx9\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:45.997470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.997429 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwh8k\" (UniqueName: \"kubernetes.io/projected/c90beec2-70e8-45fa-bcb0-055bcb27434d-kube-api-access-xwh8k\") pod \"limitador-limitador-64c94fd64f-cfmx9\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:45.997650 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.997480 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c90beec2-70e8-45fa-bcb0-055bcb27434d-config-file\") pod \"limitador-limitador-64c94fd64f-cfmx9\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:45.998135 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:45.998113 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c90beec2-70e8-45fa-bcb0-055bcb27434d-config-file\") pod \"limitador-limitador-64c94fd64f-cfmx9\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:46.005061 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.005019 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwh8k\" (UniqueName: \"kubernetes.io/projected/c90beec2-70e8-45fa-bcb0-055bcb27434d-kube-api-access-xwh8k\") pod \"limitador-limitador-64c94fd64f-cfmx9\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:46.088726 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.088689 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:46.226528 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.226453 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfmx9"] Jul 21 08:31:46.228737 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:31:46.228710 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90beec2_70e8_45fa_bcb0_055bcb27434d.slice/crio-b95914a10b2f281724fbe288005916e06d0fc3a6606923fc9b1ecff7b41db7de WatchSource:0}: Error finding container b95914a10b2f281724fbe288005916e06d0fc3a6606923fc9b1ecff7b41db7de: Status 404 returned error can't find the container with id b95914a10b2f281724fbe288005916e06d0fc3a6606923fc9b1ecff7b41db7de Jul 21 08:31:46.586550 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.586512 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-5hp7b"] Jul 21 08:31:46.589937 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.589917 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5hp7b" Jul 21 08:31:46.591653 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.591621 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" event={"ID":"c90beec2-70e8-45fa-bcb0-055bcb27434d","Type":"ContainerStarted","Data":"b95914a10b2f281724fbe288005916e06d0fc3a6606923fc9b1ecff7b41db7de"} Jul 21 08:31:46.592180 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.592163 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hkkpc\"" Jul 21 08:31:46.596290 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.596262 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5hp7b"] Jul 21 08:31:46.704925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.704897 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7g8b\" (UniqueName: \"kubernetes.io/projected/c115af3b-1279-45f7-bb18-8e8ce109dba2-kube-api-access-x7g8b\") pod \"authorino-58cd955675-5hp7b\" (UID: \"c115af3b-1279-45f7-bb18-8e8ce109dba2\") " pod="kuadrant-system/authorino-58cd955675-5hp7b" Jul 21 08:31:46.742099 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.742053 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-btt4l"] Jul 21 08:31:46.744543 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.744525 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-btt4l" Jul 21 08:31:46.752869 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.752844 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-btt4l"] Jul 21 08:31:46.805615 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.805579 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7g8b\" (UniqueName: \"kubernetes.io/projected/c115af3b-1279-45f7-bb18-8e8ce109dba2-kube-api-access-x7g8b\") pod \"authorino-58cd955675-5hp7b\" (UID: \"c115af3b-1279-45f7-bb18-8e8ce109dba2\") " pod="kuadrant-system/authorino-58cd955675-5hp7b" Jul 21 08:31:46.814026 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.813987 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7g8b\" (UniqueName: \"kubernetes.io/projected/c115af3b-1279-45f7-bb18-8e8ce109dba2-kube-api-access-x7g8b\") pod \"authorino-58cd955675-5hp7b\" (UID: \"c115af3b-1279-45f7-bb18-8e8ce109dba2\") " pod="kuadrant-system/authorino-58cd955675-5hp7b" Jul 21 08:31:46.900153 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.900067 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5hp7b" Jul 21 08:31:46.906809 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:46.906777 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwqz\" (UniqueName: \"kubernetes.io/projected/3f721c34-fdc5-41db-99be-c4468691f4ef-kube-api-access-frwqz\") pod \"authorino-75d556bd77-btt4l\" (UID: \"3f721c34-fdc5-41db-99be-c4468691f4ef\") " pod="kuadrant-system/authorino-75d556bd77-btt4l" Jul 21 08:31:47.010043 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:47.008236 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwqz\" (UniqueName: \"kubernetes.io/projected/3f721c34-fdc5-41db-99be-c4468691f4ef-kube-api-access-frwqz\") pod \"authorino-75d556bd77-btt4l\" (UID: \"3f721c34-fdc5-41db-99be-c4468691f4ef\") " pod="kuadrant-system/authorino-75d556bd77-btt4l" Jul 21 08:31:47.023368 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:47.023337 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwqz\" (UniqueName: \"kubernetes.io/projected/3f721c34-fdc5-41db-99be-c4468691f4ef-kube-api-access-frwqz\") pod \"authorino-75d556bd77-btt4l\" (UID: \"3f721c34-fdc5-41db-99be-c4468691f4ef\") " pod="kuadrant-system/authorino-75d556bd77-btt4l" Jul 21 08:31:47.053977 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:47.053936 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-btt4l" Jul 21 08:31:47.088602 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:47.086857 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5hp7b"] Jul 21 08:31:47.093941 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:31:47.093362 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc115af3b_1279_45f7_bb18_8e8ce109dba2.slice/crio-027a763d184c7bf43e4e6b02b1535619868232a410b0a941525b05490667ed5d WatchSource:0}: Error finding container 027a763d184c7bf43e4e6b02b1535619868232a410b0a941525b05490667ed5d: Status 404 returned error can't find the container with id 027a763d184c7bf43e4e6b02b1535619868232a410b0a941525b05490667ed5d Jul 21 08:31:47.251384 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:47.251353 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-btt4l"] Jul 21 08:31:47.254733 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:31:47.254685 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f721c34_fdc5_41db_99be_c4468691f4ef.slice/crio-151d2e974ef16a55f0e8c2b808bd891742974d8a308ca8888629f901dd312779 WatchSource:0}: Error finding container 151d2e974ef16a55f0e8c2b808bd891742974d8a308ca8888629f901dd312779: Status 404 returned error can't find the container with id 151d2e974ef16a55f0e8c2b808bd891742974d8a308ca8888629f901dd312779 Jul 21 08:31:47.598741 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:47.598667 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-btt4l" event={"ID":"3f721c34-fdc5-41db-99be-c4468691f4ef","Type":"ContainerStarted","Data":"151d2e974ef16a55f0e8c2b808bd891742974d8a308ca8888629f901dd312779"} Jul 21 08:31:47.599948 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:47.599919 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5hp7b" event={"ID":"c115af3b-1279-45f7-bb18-8e8ce109dba2","Type":"ContainerStarted","Data":"027a763d184c7bf43e4e6b02b1535619868232a410b0a941525b05490667ed5d"} Jul 21 08:31:51.631397 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.631347 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5hp7b" event={"ID":"c115af3b-1279-45f7-bb18-8e8ce109dba2","Type":"ContainerStarted","Data":"4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84"} Jul 21 08:31:51.632956 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.632932 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" event={"ID":"c90beec2-70e8-45fa-bcb0-055bcb27434d","Type":"ContainerStarted","Data":"0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db"} Jul 21 08:31:51.633109 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.632984 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:31:51.634389 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.634359 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-btt4l" event={"ID":"3f721c34-fdc5-41db-99be-c4468691f4ef","Type":"ContainerStarted","Data":"746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522"} Jul 21 08:31:51.646399 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.646354 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-5hp7b" podStartSLOduration=1.6639460609999999 podStartE2EDuration="5.646340897s" totalImagesPullingTime="3.982394836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:51.645427577 +0000 UTC m=+767.531098125" watchObservedRunningTime="2026-07-21 08:31:51.646340897 +0000 UTC m=+767.532011459" Jul 21 08:31:51.660748 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.660695 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-btt4l" podStartSLOduration=1.853420731 podStartE2EDuration="5.660678729s" totalImagesPullingTime="3.807257998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:51.660067664 +0000 UTC m=+767.545738216" watchObservedRunningTime="2026-07-21 08:31:51.660678729 +0000 UTC m=+767.546349278" Jul 21 08:31:51.678534 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.678475 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" podStartSLOduration=1.888844164 podStartE2EDuration="6.678459748s" totalImagesPullingTime="4.789615584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:31:51.677664656 +0000 UTC m=+767.563335204" watchObservedRunningTime="2026-07-21 08:31:51.678459748 +0000 UTC m=+767.564130297" Jul 21 08:31:51.686953 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:51.686922 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5hp7b"] Jul 21 08:31:53.644005 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:53.643961 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-5hp7b" podUID="c115af3b-1279-45f7-bb18-8e8ce109dba2" containerName="authorino" containerID="cri-o://4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84" gracePeriod=30 Jul 21 08:31:53.899983 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:53.899926 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5hp7b" Jul 21 08:31:54.072180 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.072139 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7g8b\" (UniqueName: \"kubernetes.io/projected/c115af3b-1279-45f7-bb18-8e8ce109dba2-kube-api-access-x7g8b\") pod \"c115af3b-1279-45f7-bb18-8e8ce109dba2\" (UID: \"c115af3b-1279-45f7-bb18-8e8ce109dba2\") " Jul 21 08:31:54.074281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.074257 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c115af3b-1279-45f7-bb18-8e8ce109dba2-kube-api-access-x7g8b" (OuterVolumeSpecName: "kube-api-access-x7g8b") pod "c115af3b-1279-45f7-bb18-8e8ce109dba2" (UID: "c115af3b-1279-45f7-bb18-8e8ce109dba2"). InnerVolumeSpecName "kube-api-access-x7g8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:31:54.173041 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.173007 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7g8b\" (UniqueName: \"kubernetes.io/projected/c115af3b-1279-45f7-bb18-8e8ce109dba2-kube-api-access-x7g8b\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:31:54.650100 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.650005 2553 generic.go:356] "Generic (PLEG): container finished" podID="c115af3b-1279-45f7-bb18-8e8ce109dba2" containerID="4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84" exitCode=0 Jul 21 08:31:54.650100 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.650051 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5hp7b" Jul 21 08:31:54.650643 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.650102 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5hp7b" event={"ID":"c115af3b-1279-45f7-bb18-8e8ce109dba2","Type":"ContainerDied","Data":"4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84"} Jul 21 08:31:54.650643 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.650144 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5hp7b" event={"ID":"c115af3b-1279-45f7-bb18-8e8ce109dba2","Type":"ContainerDied","Data":"027a763d184c7bf43e4e6b02b1535619868232a410b0a941525b05490667ed5d"} Jul 21 08:31:54.650643 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.650168 2553 scope.go:117] "RemoveContainer" containerID="4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84" Jul 21 08:31:54.659880 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.659865 2553 scope.go:117] "RemoveContainer" containerID="4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84" Jul 21 08:31:54.660142 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:31:54.660121 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84\": container with ID starting with 4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84 not found: ID does not exist" containerID="4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84" Jul 21 08:31:54.660211 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.660150 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84"} err="failed to get container status \"4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84\": rpc error: code = NotFound desc = could not find container \"4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84\": container with ID starting with 4342af45adf707af73bb56c3f42365a689deb52d24ed091f9149066c14453a84 not found: ID does not exist" Jul 21 08:31:54.677795 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.677766 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5hp7b"] Jul 21 08:31:54.683897 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:54.683874 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-5hp7b"] Jul 21 08:31:56.673018 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:31:56.672987 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c115af3b-1279-45f7-bb18-8e8ce109dba2" path="/var/lib/kubelet/pods/c115af3b-1279-45f7-bb18-8e8ce109dba2/volumes" Jul 21 08:32:00.532830 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:00.532793 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfmx9"] Jul 21 08:32:00.533275 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:00.532993 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" podUID="c90beec2-70e8-45fa-bcb0-055bcb27434d" containerName="limitador" containerID="cri-o://0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db" gracePeriod=30 Jul 21 08:32:00.533823 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:00.533797 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:32:01.095246 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.095189 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:32:01.133411 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.133380 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c90beec2-70e8-45fa-bcb0-055bcb27434d-config-file\") pod \"c90beec2-70e8-45fa-bcb0-055bcb27434d\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " Jul 21 08:32:01.133574 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.133438 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwh8k\" (UniqueName: \"kubernetes.io/projected/c90beec2-70e8-45fa-bcb0-055bcb27434d-kube-api-access-xwh8k\") pod \"c90beec2-70e8-45fa-bcb0-055bcb27434d\" (UID: \"c90beec2-70e8-45fa-bcb0-055bcb27434d\") " Jul 21 08:32:01.133742 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.133718 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90beec2-70e8-45fa-bcb0-055bcb27434d-config-file" (OuterVolumeSpecName: "config-file") pod "c90beec2-70e8-45fa-bcb0-055bcb27434d" (UID: "c90beec2-70e8-45fa-bcb0-055bcb27434d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:32:01.135732 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.135706 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90beec2-70e8-45fa-bcb0-055bcb27434d-kube-api-access-xwh8k" (OuterVolumeSpecName: "kube-api-access-xwh8k") pod "c90beec2-70e8-45fa-bcb0-055bcb27434d" (UID: "c90beec2-70e8-45fa-bcb0-055bcb27434d"). InnerVolumeSpecName "kube-api-access-xwh8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:32:01.234918 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.234863 2553 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c90beec2-70e8-45fa-bcb0-055bcb27434d-config-file\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:32:01.234918 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.234912 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwh8k\" (UniqueName: \"kubernetes.io/projected/c90beec2-70e8-45fa-bcb0-055bcb27434d-kube-api-access-xwh8k\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:32:01.685098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.685062 2553 generic.go:356] "Generic (PLEG): container finished" podID="c90beec2-70e8-45fa-bcb0-055bcb27434d" containerID="0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db" exitCode=0 Jul 21 08:32:01.685607 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.685119 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" Jul 21 08:32:01.685607 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.685153 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" event={"ID":"c90beec2-70e8-45fa-bcb0-055bcb27434d","Type":"ContainerDied","Data":"0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db"} Jul 21 08:32:01.685607 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.685210 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-cfmx9" event={"ID":"c90beec2-70e8-45fa-bcb0-055bcb27434d","Type":"ContainerDied","Data":"b95914a10b2f281724fbe288005916e06d0fc3a6606923fc9b1ecff7b41db7de"} Jul 21 08:32:01.685607 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.685234 2553 scope.go:117] "RemoveContainer" containerID="0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db" Jul 21 08:32:01.694174 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.694157 2553 scope.go:117] "RemoveContainer" containerID="0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db" Jul 21 08:32:01.694479 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:32:01.694459 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db\": container with ID starting with 0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db not found: ID does not exist" containerID="0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db" Jul 21 08:32:01.694551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.694492 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db"} err="failed to get container status \"0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db\": rpc error: code = NotFound desc = could not find container \"0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db\": container with ID starting with 0c20fd864be28122dcd6b28c8cbdc308cc16974c60543275be19887003f312db not found: ID does not exist" Jul 21 08:32:01.705069 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.705042 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfmx9"] Jul 21 08:32:01.708764 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:01.708743 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-cfmx9"] Jul 21 08:32:02.443007 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.442972 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zlt99"] Jul 21 08:32:02.443341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.443322 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90beec2-70e8-45fa-bcb0-055bcb27434d" containerName="limitador" Jul 21 08:32:02.443341 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.443336 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90beec2-70e8-45fa-bcb0-055bcb27434d" containerName="limitador" Jul 21 08:32:02.443506 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.443352 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c115af3b-1279-45f7-bb18-8e8ce109dba2" containerName="authorino" Jul 21 08:32:02.443506 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.443358 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c115af3b-1279-45f7-bb18-8e8ce109dba2" containerName="authorino" Jul 21 08:32:02.443506 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.443420 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90beec2-70e8-45fa-bcb0-055bcb27434d" containerName="limitador" Jul 21 08:32:02.443506 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.443430 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c115af3b-1279-45f7-bb18-8e8ce109dba2" containerName="authorino" Jul 21 08:32:02.447587 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.447567 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.450053 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.450033 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 08:32:02.450866 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.450849 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 08:32:02.450993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.450867 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 08:32:02.450993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.450875 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wbwg7\"" Jul 21 08:32:02.457724 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.457224 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zlt99"] Jul 21 08:32:02.546298 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.546258 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2c87943-9a89-4b9e-a8d4-f9b4f1909890-data\") pod \"postgres-69fb4594cf-zlt99\" (UID: \"e2c87943-9a89-4b9e-a8d4-f9b4f1909890\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.546531 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.546410 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrrc\" (UniqueName: \"kubernetes.io/projected/e2c87943-9a89-4b9e-a8d4-f9b4f1909890-kube-api-access-znrrc\") pod \"postgres-69fb4594cf-zlt99\" (UID: \"e2c87943-9a89-4b9e-a8d4-f9b4f1909890\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.647397 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.647358 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2c87943-9a89-4b9e-a8d4-f9b4f1909890-data\") pod \"postgres-69fb4594cf-zlt99\" (UID: \"e2c87943-9a89-4b9e-a8d4-f9b4f1909890\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.647602 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.647424 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrrc\" (UniqueName: \"kubernetes.io/projected/e2c87943-9a89-4b9e-a8d4-f9b4f1909890-kube-api-access-znrrc\") pod \"postgres-69fb4594cf-zlt99\" (UID: \"e2c87943-9a89-4b9e-a8d4-f9b4f1909890\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.647801 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.647776 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2c87943-9a89-4b9e-a8d4-f9b4f1909890-data\") pod \"postgres-69fb4594cf-zlt99\" (UID: \"e2c87943-9a89-4b9e-a8d4-f9b4f1909890\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.659877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.659846 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrrc\" (UniqueName: \"kubernetes.io/projected/e2c87943-9a89-4b9e-a8d4-f9b4f1909890-kube-api-access-znrrc\") pod \"postgres-69fb4594cf-zlt99\" (UID: \"e2c87943-9a89-4b9e-a8d4-f9b4f1909890\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.674697 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.674663 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90beec2-70e8-45fa-bcb0-055bcb27434d" path="/var/lib/kubelet/pods/c90beec2-70e8-45fa-bcb0-055bcb27434d/volumes" Jul 21 08:32:02.765649 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.765565 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:02.901522 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:02.901498 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zlt99"] Jul 21 08:32:02.903818 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:32:02.903792 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c87943_9a89_4b9e_a8d4_f9b4f1909890.slice/crio-ab5c470c0f8e90c4ca4ca068f86ceb35b35d2ed7cda23c4bffefff1190238548 WatchSource:0}: Error finding container ab5c470c0f8e90c4ca4ca068f86ceb35b35d2ed7cda23c4bffefff1190238548: Status 404 returned error can't find the container with id ab5c470c0f8e90c4ca4ca068f86ceb35b35d2ed7cda23c4bffefff1190238548 Jul 21 08:32:03.697620 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:03.697586 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" event={"ID":"e2c87943-9a89-4b9e-a8d4-f9b4f1909890","Type":"ContainerStarted","Data":"ab5c470c0f8e90c4ca4ca068f86ceb35b35d2ed7cda23c4bffefff1190238548"} Jul 21 08:32:08.079327 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:08.079291 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 08:32:08.729864 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:08.729823 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" event={"ID":"e2c87943-9a89-4b9e-a8d4-f9b4f1909890","Type":"ContainerStarted","Data":"d6b9c7bbb9ff4f34ffd9d79c61b14fa99b83c30d6b6d471c872ce2b0a4add605"} Jul 21 08:32:08.730082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:08.729926 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:08.748283 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:08.748224 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" podStartSLOduration=1.5765289729999998 podStartE2EDuration="6.748205846s" totalImagesPullingTime="5.171676873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:32:08.746238405 +0000 UTC m=+784.631908955" watchObservedRunningTime="2026-07-21 08:32:08.748205846 +0000 UTC m=+784.633876393" Jul 21 08:32:14.768156 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:14.768127 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zlt99" Jul 21 08:32:15.323332 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.323274 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8ct59"] Jul 21 08:32:15.328831 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.328810 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8ct59" Jul 21 08:32:15.333699 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.333673 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8ct59"] Jul 21 08:32:15.468704 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.468663 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcq8c\" (UniqueName: \"kubernetes.io/projected/7e0474b0-6c82-475e-ba53-94be86edc36e-kube-api-access-pcq8c\") pod \"authorino-5d4b67c446-8ct59\" (UID: \"7e0474b0-6c82-475e-ba53-94be86edc36e\") " pod="kuadrant-system/authorino-5d4b67c446-8ct59" Jul 21 08:32:15.569459 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.569417 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcq8c\" (UniqueName: \"kubernetes.io/projected/7e0474b0-6c82-475e-ba53-94be86edc36e-kube-api-access-pcq8c\") pod \"authorino-5d4b67c446-8ct59\" (UID: \"7e0474b0-6c82-475e-ba53-94be86edc36e\") " pod="kuadrant-system/authorino-5d4b67c446-8ct59" Jul 21 08:32:15.575132 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.575067 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8ct59"] Jul 21 08:32:15.575378 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:32:15.575354 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pcq8c], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-8ct59" podUID="7e0474b0-6c82-475e-ba53-94be86edc36e" Jul 21 08:32:15.577956 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.577937 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcq8c\" (UniqueName: \"kubernetes.io/projected/7e0474b0-6c82-475e-ba53-94be86edc36e-kube-api-access-pcq8c\") pod \"authorino-5d4b67c446-8ct59\" (UID: \"7e0474b0-6c82-475e-ba53-94be86edc36e\") " pod="kuadrant-system/authorino-5d4b67c446-8ct59" Jul 21 08:32:15.605488 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.605455 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f45f6fb66-mhhm7"] Jul 21 08:32:15.609393 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.609374 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" Jul 21 08:32:15.614454 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.614430 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f45f6fb66-mhhm7"] Jul 21 08:32:15.670665 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.670630 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g68\" (UniqueName: \"kubernetes.io/projected/f25a1585-1853-47ed-9f70-a3c28a35e503-kube-api-access-g6g68\") pod \"authorino-6f45f6fb66-mhhm7\" (UID: \"f25a1585-1853-47ed-9f70-a3c28a35e503\") " pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" Jul 21 08:32:15.763758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.763727 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8ct59" Jul 21 08:32:15.770734 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.770714 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8ct59" Jul 21 08:32:15.771094 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.771079 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6g68\" (UniqueName: \"kubernetes.io/projected/f25a1585-1853-47ed-9f70-a3c28a35e503-kube-api-access-g6g68\") pod \"authorino-6f45f6fb66-mhhm7\" (UID: \"f25a1585-1853-47ed-9f70-a3c28a35e503\") " pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" Jul 21 08:32:15.774291 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.774255 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f45f6fb66-mhhm7"] Jul 21 08:32:15.774554 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:32:15.774530 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g6g68], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" podUID="f25a1585-1853-47ed-9f70-a3c28a35e503" Jul 21 08:32:15.787427 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.787404 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6g68\" (UniqueName: \"kubernetes.io/projected/f25a1585-1853-47ed-9f70-a3c28a35e503-kube-api-access-g6g68\") pod \"authorino-6f45f6fb66-mhhm7\" (UID: \"f25a1585-1853-47ed-9f70-a3c28a35e503\") " pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" Jul 21 08:32:15.804489 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.804463 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-799d4bf97d-8x8q5"] Jul 21 08:32:15.807964 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.807946 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:15.810160 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.810137 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 08:32:15.814895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.814867 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-799d4bf97d-8x8q5"] Jul 21 08:32:15.872048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.871950 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcq8c\" (UniqueName: \"kubernetes.io/projected/7e0474b0-6c82-475e-ba53-94be86edc36e-kube-api-access-pcq8c\") pod \"7e0474b0-6c82-475e-ba53-94be86edc36e\" (UID: \"7e0474b0-6c82-475e-ba53-94be86edc36e\") " Jul 21 08:32:15.872228 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.872128 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b48673b-b50e-4a28-94ec-c130b7dec4b7-tls-cert\") pod \"authorino-799d4bf97d-8x8q5\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:15.872228 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.872191 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmrc\" (UniqueName: \"kubernetes.io/projected/8b48673b-b50e-4a28-94ec-c130b7dec4b7-kube-api-access-bbmrc\") pod \"authorino-799d4bf97d-8x8q5\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:15.874266 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.874236 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0474b0-6c82-475e-ba53-94be86edc36e-kube-api-access-pcq8c" (OuterVolumeSpecName: "kube-api-access-pcq8c") pod "7e0474b0-6c82-475e-ba53-94be86edc36e" (UID: "7e0474b0-6c82-475e-ba53-94be86edc36e"). InnerVolumeSpecName "kube-api-access-pcq8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:32:15.973657 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.973620 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b48673b-b50e-4a28-94ec-c130b7dec4b7-tls-cert\") pod \"authorino-799d4bf97d-8x8q5\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:15.973835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.973678 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmrc\" (UniqueName: \"kubernetes.io/projected/8b48673b-b50e-4a28-94ec-c130b7dec4b7-kube-api-access-bbmrc\") pod \"authorino-799d4bf97d-8x8q5\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:15.973835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.973732 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcq8c\" (UniqueName: \"kubernetes.io/projected/7e0474b0-6c82-475e-ba53-94be86edc36e-kube-api-access-pcq8c\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:32:15.976185 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.976167 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b48673b-b50e-4a28-94ec-c130b7dec4b7-tls-cert\") pod \"authorino-799d4bf97d-8x8q5\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:15.982420 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:15.982390 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmrc\" (UniqueName: \"kubernetes.io/projected/8b48673b-b50e-4a28-94ec-c130b7dec4b7-kube-api-access-bbmrc\") pod \"authorino-799d4bf97d-8x8q5\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:16.118845 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.118803 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:32:16.250972 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.250942 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-799d4bf97d-8x8q5"] Jul 21 08:32:16.253432 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:32:16.253407 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b48673b_b50e_4a28_94ec_c130b7dec4b7.slice/crio-55e6086a3580d388d53437d43130f6dcae981a17fac62313bcf536c5000c437d WatchSource:0}: Error finding container 55e6086a3580d388d53437d43130f6dcae981a17fac62313bcf536c5000c437d: Status 404 returned error can't find the container with id 55e6086a3580d388d53437d43130f6dcae981a17fac62313bcf536c5000c437d Jul 21 08:32:16.769624 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.769592 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" Jul 21 08:32:16.769624 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.769625 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-8ct59" Jul 21 08:32:16.769838 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.769594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" event={"ID":"8b48673b-b50e-4a28-94ec-c130b7dec4b7","Type":"ContainerStarted","Data":"55e6086a3580d388d53437d43130f6dcae981a17fac62313bcf536c5000c437d"} Jul 21 08:32:16.778604 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.778575 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" Jul 21 08:32:16.808676 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.808644 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8ct59"] Jul 21 08:32:16.812048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.812023 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-8ct59"] Jul 21 08:32:16.882000 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.881964 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6g68\" (UniqueName: \"kubernetes.io/projected/f25a1585-1853-47ed-9f70-a3c28a35e503-kube-api-access-g6g68\") pod \"f25a1585-1853-47ed-9f70-a3c28a35e503\" (UID: \"f25a1585-1853-47ed-9f70-a3c28a35e503\") " Jul 21 08:32:16.884165 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.884139 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25a1585-1853-47ed-9f70-a3c28a35e503-kube-api-access-g6g68" (OuterVolumeSpecName: "kube-api-access-g6g68") pod "f25a1585-1853-47ed-9f70-a3c28a35e503" (UID: "f25a1585-1853-47ed-9f70-a3c28a35e503"). InnerVolumeSpecName "kube-api-access-g6g68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:32:16.983522 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:16.983437 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6g68\" (UniqueName: \"kubernetes.io/projected/f25a1585-1853-47ed-9f70-a3c28a35e503-kube-api-access-g6g68\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:32:17.774158 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:17.774130 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f45f6fb66-mhhm7" Jul 21 08:32:17.800546 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:17.800512 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f45f6fb66-mhhm7"] Jul 21 08:32:17.802201 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:17.802180 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f45f6fb66-mhhm7"] Jul 21 08:32:18.673566 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:18.673524 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0474b0-6c82-475e-ba53-94be86edc36e" path="/var/lib/kubelet/pods/7e0474b0-6c82-475e-ba53-94be86edc36e/volumes" Jul 21 08:32:18.673758 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:18.673746 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25a1585-1853-47ed-9f70-a3c28a35e503" path="/var/lib/kubelet/pods/f25a1585-1853-47ed-9f70-a3c28a35e503/volumes" Jul 21 08:32:19.784455 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:19.784406 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" event={"ID":"8b48673b-b50e-4a28-94ec-c130b7dec4b7","Type":"ContainerStarted","Data":"15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0"} Jul 21 08:32:19.799718 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:19.799660 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" podStartSLOduration=1.698833714 podStartE2EDuration="4.799643538s" totalImagesPullingTime="3.100809824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:32:19.797647565 +0000 UTC m=+795.683318111" watchObservedRunningTime="2026-07-21 08:32:19.799643538 +0000 UTC m=+795.685314116" Jul 21 08:32:19.822725 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:19.822691 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-btt4l"] Jul 21 08:32:19.822899 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:19.822872 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-btt4l" podUID="3f721c34-fdc5-41db-99be-c4468691f4ef" containerName="authorino" containerID="cri-o://746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522" gracePeriod=30 Jul 21 08:32:20.092848 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.092821 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-btt4l" Jul 21 08:32:20.211005 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.210974 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frwqz\" (UniqueName: \"kubernetes.io/projected/3f721c34-fdc5-41db-99be-c4468691f4ef-kube-api-access-frwqz\") pod \"3f721c34-fdc5-41db-99be-c4468691f4ef\" (UID: \"3f721c34-fdc5-41db-99be-c4468691f4ef\") " Jul 21 08:32:20.213163 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.213127 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f721c34-fdc5-41db-99be-c4468691f4ef-kube-api-access-frwqz" (OuterVolumeSpecName: "kube-api-access-frwqz") pod "3f721c34-fdc5-41db-99be-c4468691f4ef" (UID: "3f721c34-fdc5-41db-99be-c4468691f4ef"). InnerVolumeSpecName "kube-api-access-frwqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:32:20.312411 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.312336 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwqz\" (UniqueName: \"kubernetes.io/projected/3f721c34-fdc5-41db-99be-c4468691f4ef-kube-api-access-frwqz\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:32:20.789857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.789823 2553 generic.go:356] "Generic (PLEG): container finished" podID="3f721c34-fdc5-41db-99be-c4468691f4ef" containerID="746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522" exitCode=0 Jul 21 08:32:20.790330 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.789861 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-btt4l" Jul 21 08:32:20.790330 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.789909 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-btt4l" event={"ID":"3f721c34-fdc5-41db-99be-c4468691f4ef","Type":"ContainerDied","Data":"746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522"} Jul 21 08:32:20.790330 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.789960 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-btt4l" event={"ID":"3f721c34-fdc5-41db-99be-c4468691f4ef","Type":"ContainerDied","Data":"151d2e974ef16a55f0e8c2b808bd891742974d8a308ca8888629f901dd312779"} Jul 21 08:32:20.790330 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.789979 2553 scope.go:117] "RemoveContainer" containerID="746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522" Jul 21 08:32:20.798491 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.798468 2553 scope.go:117] "RemoveContainer" containerID="746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522" Jul 21 08:32:20.798723 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:32:20.798704 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522\": container with ID starting with 746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522 not found: ID does not exist" containerID="746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522" Jul 21 08:32:20.798781 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.798730 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522"} err="failed to get container status \"746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522\": rpc error: code = NotFound desc = could not find container \"746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522\": container with ID starting with 746437b591dda2a304ff68bfcc2bf0488d995b7dfad4de3060cd2f33ac1c0522 not found: ID does not exist" Jul 21 08:32:20.806039 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.806016 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-btt4l"] Jul 21 08:32:20.810020 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:20.809996 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-btt4l"] Jul 21 08:32:22.674545 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:22.674512 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f721c34-fdc5-41db-99be-c4468691f4ef" path="/var/lib/kubelet/pods/3f721c34-fdc5-41db-99be-c4468691f4ef/volumes" Jul 21 08:32:24.630055 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.629976 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d58d7c5f8-l8p7h"] Jul 21 08:32:24.630529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.630343 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f721c34-fdc5-41db-99be-c4468691f4ef" containerName="authorino" Jul 21 08:32:24.630529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.630355 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f721c34-fdc5-41db-99be-c4468691f4ef" containerName="authorino" Jul 21 08:32:24.630529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.630422 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f721c34-fdc5-41db-99be-c4468691f4ef" containerName="authorino" Jul 21 08:32:24.647398 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.647365 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d58d7c5f8-l8p7h"] Jul 21 08:32:24.647577 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.647500 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:24.651017 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.650988 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 08:32:24.651411 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.651380 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zmxb6\"" Jul 21 08:32:24.651952 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.651919 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 08:32:24.750936 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.750903 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert\") pod \"maas-controller-7d58d7c5f8-l8p7h\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:24.751117 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.751072 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbdwf\" (UniqueName: \"kubernetes.io/projected/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-kube-api-access-mbdwf\") pod \"maas-controller-7d58d7c5f8-l8p7h\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:24.815372 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.815338 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4hjl9"] Jul 21 08:32:24.817945 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.817926 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:24.829557 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.829535 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4hjl9"] Jul 21 08:32:24.851699 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.851670 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert\") pod \"maas-controller-7d58d7c5f8-l8p7h\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:24.851848 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.851784 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbdwf\" (UniqueName: \"kubernetes.io/projected/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-kube-api-access-mbdwf\") pod \"maas-controller-7d58d7c5f8-l8p7h\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:24.851895 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:32:24.851849 2553 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 21 08:32:24.851956 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:32:24.851944 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert podName:3b0aba3d-890d-4b84-8879-e2a0b5cf40f6 nodeName:}" failed. No retries permitted until 2026-07-21 08:32:25.351917417 +0000 UTC m=+801.237587969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert") pod "maas-controller-7d58d7c5f8-l8p7h" (UID: "3b0aba3d-890d-4b84-8879-e2a0b5cf40f6") : secret "maas-controller-webhook-cert" not found Jul 21 08:32:24.860262 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.860239 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbdwf\" (UniqueName: \"kubernetes.io/projected/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-kube-api-access-mbdwf\") pod \"maas-controller-7d58d7c5f8-l8p7h\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:24.953159 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.953106 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btxz9\" (UniqueName: \"kubernetes.io/projected/f38df518-35a9-461b-a365-96f3bb967902-kube-api-access-btxz9\") pod \"maas-controller-68d799f9c8-4hjl9\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:24.953392 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:24.953259 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f38df518-35a9-461b-a365-96f3bb967902-webhook-cert\") pod \"maas-controller-68d799f9c8-4hjl9\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:25.054546 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.054498 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btxz9\" (UniqueName: \"kubernetes.io/projected/f38df518-35a9-461b-a365-96f3bb967902-kube-api-access-btxz9\") pod \"maas-controller-68d799f9c8-4hjl9\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:25.054769 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.054650 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f38df518-35a9-461b-a365-96f3bb967902-webhook-cert\") pod \"maas-controller-68d799f9c8-4hjl9\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:25.057377 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.057349 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f38df518-35a9-461b-a365-96f3bb967902-webhook-cert\") pod \"maas-controller-68d799f9c8-4hjl9\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:25.062779 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.062754 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btxz9\" (UniqueName: \"kubernetes.io/projected/f38df518-35a9-461b-a365-96f3bb967902-kube-api-access-btxz9\") pod \"maas-controller-68d799f9c8-4hjl9\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:25.128639 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.128599 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:25.262890 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.262844 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4hjl9"] Jul 21 08:32:25.265088 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:32:25.265057 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38df518_35a9_461b_a365_96f3bb967902.slice/crio-0d654fcac293240874fc9e4cf56b70ecf2e06e4caf5cab6cf81ffe336cef53c7 WatchSource:0}: Error finding container 0d654fcac293240874fc9e4cf56b70ecf2e06e4caf5cab6cf81ffe336cef53c7: Status 404 returned error can't find the container with id 0d654fcac293240874fc9e4cf56b70ecf2e06e4caf5cab6cf81ffe336cef53c7 Jul 21 08:32:25.357717 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.357684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert\") pod \"maas-controller-7d58d7c5f8-l8p7h\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:25.360180 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.360162 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert\") pod \"maas-controller-7d58d7c5f8-l8p7h\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:25.560390 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.560286 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:25.692613 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.692589 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d58d7c5f8-l8p7h"] Jul 21 08:32:25.694714 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:32:25.694686 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0aba3d_890d_4b84_8879_e2a0b5cf40f6.slice/crio-e41dbd0d0c3339130f8a17a1206c7e5834ff70ba8bcf0c1d0ca421cfa4993208 WatchSource:0}: Error finding container e41dbd0d0c3339130f8a17a1206c7e5834ff70ba8bcf0c1d0ca421cfa4993208: Status 404 returned error can't find the container with id e41dbd0d0c3339130f8a17a1206c7e5834ff70ba8bcf0c1d0ca421cfa4993208 Jul 21 08:32:25.815015 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.814917 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" event={"ID":"f38df518-35a9-461b-a365-96f3bb967902","Type":"ContainerStarted","Data":"0d654fcac293240874fc9e4cf56b70ecf2e06e4caf5cab6cf81ffe336cef53c7"} Jul 21 08:32:25.816046 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:25.816025 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" event={"ID":"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6","Type":"ContainerStarted","Data":"e41dbd0d0c3339130f8a17a1206c7e5834ff70ba8bcf0c1d0ca421cfa4993208"} Jul 21 08:32:28.838843 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:28.838728 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" event={"ID":"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6","Type":"ContainerStarted","Data":"d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a"} Jul 21 08:32:28.838843 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:28.838810 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:28.840301 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:28.840274 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" event={"ID":"f38df518-35a9-461b-a365-96f3bb967902","Type":"ContainerStarted","Data":"fc2160665d92720f670dbfed8be6b4909cd233f4c6306c89dc91eccc00b56aea"} Jul 21 08:32:28.840399 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:28.840357 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:28.858939 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:28.858870 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" podStartSLOduration=2.434948385 podStartE2EDuration="4.858855184s" totalImagesPullingTime="2.423906799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:32:28.857626846 +0000 UTC m=+804.743297416" watchObservedRunningTime="2026-07-21 08:32:28.858855184 +0000 UTC m=+804.744525892" Jul 21 08:32:28.881187 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:28.881138 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" podStartSLOduration=1.63005297 podStartE2EDuration="4.881120777s" totalImagesPullingTime="3.251067807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:32:28.880570075 +0000 UTC m=+804.766240624" watchObservedRunningTime="2026-07-21 08:32:28.881120777 +0000 UTC m=+804.766791325" Jul 21 08:32:39.847725 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:39.847689 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:32:39.848182 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:39.847770 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:39.915063 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:39.915022 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d58d7c5f8-l8p7h"] Jul 21 08:32:39.915277 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:39.915249 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" podUID="3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" containerName="manager" containerID="cri-o://d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a" gracePeriod=30 Jul 21 08:32:40.168112 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.168087 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:40.296173 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.296142 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert\") pod \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " Jul 21 08:32:40.296438 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.296288 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbdwf\" (UniqueName: \"kubernetes.io/projected/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-kube-api-access-mbdwf\") pod \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\" (UID: \"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6\") " Jul 21 08:32:40.298515 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.298480 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-kube-api-access-mbdwf" (OuterVolumeSpecName: "kube-api-access-mbdwf") pod "3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" (UID: "3b0aba3d-890d-4b84-8879-e2a0b5cf40f6"). InnerVolumeSpecName "kube-api-access-mbdwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:32:40.298602 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.298566 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" (UID: "3b0aba3d-890d-4b84-8879-e2a0b5cf40f6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:32:40.397172 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.397135 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-webhook-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:32:40.397172 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.397166 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbdwf\" (UniqueName: \"kubernetes.io/projected/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6-kube-api-access-mbdwf\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:32:40.899557 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.899527 2553 generic.go:356] "Generic (PLEG): container finished" podID="3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" containerID="d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a" exitCode=0 Jul 21 08:32:40.899957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.899572 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" Jul 21 08:32:40.899957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.899622 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" event={"ID":"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6","Type":"ContainerDied","Data":"d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a"} Jul 21 08:32:40.899957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.899675 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d58d7c5f8-l8p7h" event={"ID":"3b0aba3d-890d-4b84-8879-e2a0b5cf40f6","Type":"ContainerDied","Data":"e41dbd0d0c3339130f8a17a1206c7e5834ff70ba8bcf0c1d0ca421cfa4993208"} Jul 21 08:32:40.899957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.899698 2553 scope.go:117] "RemoveContainer" containerID="d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a" Jul 21 08:32:40.908440 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.908421 2553 scope.go:117] "RemoveContainer" containerID="d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a" Jul 21 08:32:40.908713 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:32:40.908693 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a\": container with ID starting with d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a not found: ID does not exist" containerID="d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a" Jul 21 08:32:40.908788 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.908726 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a"} err="failed to get container status \"d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a\": rpc error: code = NotFound desc = could not find container \"d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a\": container with ID starting with d3e0a1a3ce720ac0f1330e67429bb06178829d469a1ae2be17b4e24d023c7e0a not found: ID does not exist" Jul 21 08:32:40.916283 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.916257 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d58d7c5f8-l8p7h"] Jul 21 08:32:40.922489 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:40.922469 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d58d7c5f8-l8p7h"] Jul 21 08:32:42.674174 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:32:42.674136 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" path="/var/lib/kubelet/pods/3b0aba3d-890d-4b84-8879-e2a0b5cf40f6/volumes" Jul 21 08:33:00.964517 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.964480 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr"] Jul 21 08:33:00.964902 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.964828 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" containerName="manager" Jul 21 08:33:00.964902 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.964839 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" containerName="manager" Jul 21 08:33:00.964979 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.964913 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0aba3d-890d-4b84-8879-e2a0b5cf40f6" containerName="manager" Jul 21 08:33:00.969876 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.969856 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:00.971957 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.971935 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 08:33:00.972083 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.972044 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h9jl5\"" Jul 21 08:33:00.978852 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:00.978830 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr"] Jul 21 08:33:01.069958 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.069926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cf2062-682e-4e5c-bad8-763f624e557d-openshift-service-ca\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.070129 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.069966 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfmnx\" (UniqueName: \"kubernetes.io/projected/49cf2062-682e-4e5c-bad8-763f624e557d-kube-api-access-lfmnx\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.070129 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.070056 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cf2062-682e-4e5c-bad8-763f624e557d-maas-api-tls\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.171063 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.171029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cf2062-682e-4e5c-bad8-763f624e557d-maas-api-tls\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.171267 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.171127 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cf2062-682e-4e5c-bad8-763f624e557d-openshift-service-ca\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.171267 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.171162 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfmnx\" (UniqueName: \"kubernetes.io/projected/49cf2062-682e-4e5c-bad8-763f624e557d-kube-api-access-lfmnx\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.171950 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.171923 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cf2062-682e-4e5c-bad8-763f624e557d-openshift-service-ca\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.173886 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.173863 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cf2062-682e-4e5c-bad8-763f624e557d-maas-api-tls\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.179018 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.178993 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfmnx\" (UniqueName: \"kubernetes.io/projected/49cf2062-682e-4e5c-bad8-763f624e557d-kube-api-access-lfmnx\") pod \"maas-api-86f46b5fc6-swpsr\" (UID: \"49cf2062-682e-4e5c-bad8-763f624e557d\") " pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.282123 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.282042 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:01.422845 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.422814 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bmsts"] Jul 21 08:33:01.426907 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.426820 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.427240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.427218 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr"] Jul 21 08:33:01.429280 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.429258 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-45krr\"" Jul 21 08:33:01.429416 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.429260 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 08:33:01.430004 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:33:01.429984 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cf2062_682e_4e5c_bad8_763f624e557d.slice/crio-0103eaf6059ea15fcc527f24d903a844bda04aee20a178c6709c30fcf1c96add WatchSource:0}: Error finding container 0103eaf6059ea15fcc527f24d903a844bda04aee20a178c6709c30fcf1c96add: Status 404 returned error can't find the container with id 0103eaf6059ea15fcc527f24d903a844bda04aee20a178c6709c30fcf1c96add Jul 21 08:33:01.435790 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.435766 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bmsts"] Jul 21 08:33:01.450130 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.450105 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-ltznq"] Jul 21 08:33:01.453063 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.453043 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.461161 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.461130 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-ltznq"] Jul 21 08:33:01.473584 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.473551 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45f1607-5111-4a02-953e-64a20cda6cd3-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bmsts\" (UID: \"c45f1607-5111-4a02-953e-64a20cda6cd3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.473759 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.473624 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2j4\" (UniqueName: \"kubernetes.io/projected/c45f1607-5111-4a02-953e-64a20cda6cd3-kube-api-access-wg2j4\") pod \"payload-pre-processing-7677b774bb-bmsts\" (UID: \"c45f1607-5111-4a02-953e-64a20cda6cd3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.574694 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.574603 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2j4\" (UniqueName: \"kubernetes.io/projected/c45f1607-5111-4a02-953e-64a20cda6cd3-kube-api-access-wg2j4\") pod \"payload-pre-processing-7677b774bb-bmsts\" (UID: \"c45f1607-5111-4a02-953e-64a20cda6cd3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.574694 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.574657 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51bae1-88cf-461b-b19e-4395574fba30-plugins-config-volume\") pod \"payload-processing-5bf47cd988-ltznq\" (UID: \"3e51bae1-88cf-461b-b19e-4395574fba30\") " pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.574925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.574727 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45f1607-5111-4a02-953e-64a20cda6cd3-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bmsts\" (UID: \"c45f1607-5111-4a02-953e-64a20cda6cd3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.574925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.574745 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8c6g\" (UniqueName: \"kubernetes.io/projected/3e51bae1-88cf-461b-b19e-4395574fba30-kube-api-access-g8c6g\") pod \"payload-processing-5bf47cd988-ltznq\" (UID: \"3e51bae1-88cf-461b-b19e-4395574fba30\") " pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.575460 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.575436 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c45f1607-5111-4a02-953e-64a20cda6cd3-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bmsts\" (UID: \"c45f1607-5111-4a02-953e-64a20cda6cd3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.584101 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.584076 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2j4\" (UniqueName: \"kubernetes.io/projected/c45f1607-5111-4a02-953e-64a20cda6cd3-kube-api-access-wg2j4\") pod \"payload-pre-processing-7677b774bb-bmsts\" (UID: \"c45f1607-5111-4a02-953e-64a20cda6cd3\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.675673 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.675634 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51bae1-88cf-461b-b19e-4395574fba30-plugins-config-volume\") pod \"payload-processing-5bf47cd988-ltznq\" (UID: \"3e51bae1-88cf-461b-b19e-4395574fba30\") " pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.675861 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.675757 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8c6g\" (UniqueName: \"kubernetes.io/projected/3e51bae1-88cf-461b-b19e-4395574fba30-kube-api-access-g8c6g\") pod \"payload-processing-5bf47cd988-ltznq\" (UID: \"3e51bae1-88cf-461b-b19e-4395574fba30\") " pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.676517 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.676496 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e51bae1-88cf-461b-b19e-4395574fba30-plugins-config-volume\") pod \"payload-processing-5bf47cd988-ltznq\" (UID: \"3e51bae1-88cf-461b-b19e-4395574fba30\") " pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.686843 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.686815 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8c6g\" (UniqueName: \"kubernetes.io/projected/3e51bae1-88cf-461b-b19e-4395574fba30-kube-api-access-g8c6g\") pod \"payload-processing-5bf47cd988-ltznq\" (UID: \"3e51bae1-88cf-461b-b19e-4395574fba30\") " pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.749028 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.748990 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:01.765268 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.765239 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:01.907286 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.907240 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bmsts"] Jul 21 08:33:01.909078 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:33:01.909052 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45f1607_5111_4a02_953e_64a20cda6cd3.slice/crio-92f81bde12ccadb94a75da89b8ce9da2695bc056fe8da5894a3ffc7c2c7e8d44 WatchSource:0}: Error finding container 92f81bde12ccadb94a75da89b8ce9da2695bc056fe8da5894a3ffc7c2c7e8d44: Status 404 returned error can't find the container with id 92f81bde12ccadb94a75da89b8ce9da2695bc056fe8da5894a3ffc7c2c7e8d44 Jul 21 08:33:01.923158 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:01.923131 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-ltznq"] Jul 21 08:33:01.924963 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:33:01.924929 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e51bae1_88cf_461b_b19e_4395574fba30.slice/crio-5bf98fdfadf9fa47f4655aed08b4aff5fac3158ec4d89939f02731ede02dcf24 WatchSource:0}: Error finding container 5bf98fdfadf9fa47f4655aed08b4aff5fac3158ec4d89939f02731ede02dcf24: Status 404 returned error can't find the container with id 5bf98fdfadf9fa47f4655aed08b4aff5fac3158ec4d89939f02731ede02dcf24 Jul 21 08:33:02.000655 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:02.000616 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" event={"ID":"49cf2062-682e-4e5c-bad8-763f624e557d","Type":"ContainerStarted","Data":"0103eaf6059ea15fcc527f24d903a844bda04aee20a178c6709c30fcf1c96add"} Jul 21 08:33:02.002013 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:02.001979 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" event={"ID":"c45f1607-5111-4a02-953e-64a20cda6cd3","Type":"ContainerStarted","Data":"92f81bde12ccadb94a75da89b8ce9da2695bc056fe8da5894a3ffc7c2c7e8d44"} Jul 21 08:33:02.003522 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:02.003494 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" event={"ID":"3e51bae1-88cf-461b-b19e-4395574fba30","Type":"ContainerStarted","Data":"5bf98fdfadf9fa47f4655aed08b4aff5fac3158ec4d89939f02731ede02dcf24"} Jul 21 08:33:04.025247 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:04.025208 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" event={"ID":"49cf2062-682e-4e5c-bad8-763f624e557d","Type":"ContainerStarted","Data":"d4640d1968e43b089a6ce5111e4a2d502567fa804c3d2d8f0f07894ae59d2cb1"} Jul 21 08:33:04.025709 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:04.025359 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:04.042662 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:04.042608 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" podStartSLOduration=2.149331447 podStartE2EDuration="4.042588688s" totalImagesPullingTime="1.893257241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:33:04.040658422 +0000 UTC m=+839.926328989" watchObservedRunningTime="2026-07-21 08:33:04.042588688 +0000 UTC m=+839.928259235" Jul 21 08:33:06.038734 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:06.038696 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" event={"ID":"c45f1607-5111-4a02-953e-64a20cda6cd3","Type":"ContainerStarted","Data":"f9237b684308bf692d53f000d6c3ccf811c0d2e5e56823a63b007424210bac2a"} Jul 21 08:33:06.039198 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:06.038776 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:06.054267 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:06.054216 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" podStartSLOduration=1.83173079 podStartE2EDuration="5.054199798s" totalImagesPullingTime="3.222469008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:33:06.052902389 +0000 UTC m=+841.938572938" watchObservedRunningTime="2026-07-21 08:33:06.054199798 +0000 UTC m=+841.939870751" Jul 21 08:33:07.046175 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:07.046141 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" event={"ID":"3e51bae1-88cf-461b-b19e-4395574fba30","Type":"ContainerStarted","Data":"ee17ac65b211372068ed246cf8a0d0b26a99ade23e03fd26c031e9acf51a17af"} Jul 21 08:33:07.046632 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:07.046224 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:07.062053 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:07.062000 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" podStartSLOduration=1.7697046429999999 podStartE2EDuration="6.061981639s" totalImagesPullingTime="4.292276996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:33:07.061369069 +0000 UTC m=+842.947039615" watchObservedRunningTime="2026-07-21 08:33:07.061981639 +0000 UTC m=+842.947652217" Jul 21 08:33:10.036732 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:10.036704 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86f46b5fc6-swpsr" Jul 21 08:33:10.062010 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:10.061969 2553 generic.go:356] "Generic (PLEG): container finished" podID="9eb90821-604b-40ab-8efb-f82ca783417a" containerID="c422f34fbde2e0ace5953dafbc9126f626d91ac2b53aee09b3e968844cff0c65" exitCode=0 Jul 21 08:33:10.062190 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:10.062018 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" event={"ID":"9eb90821-604b-40ab-8efb-f82ca783417a","Type":"ContainerDied","Data":"c422f34fbde2e0ace5953dafbc9126f626d91ac2b53aee09b3e968844cff0c65"} Jul 21 08:33:10.062460 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:10.062444 2553 scope.go:117] "RemoveContainer" containerID="c422f34fbde2e0ace5953dafbc9126f626d91ac2b53aee09b3e968844cff0c65" Jul 21 08:33:10.063493 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:10.063452 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:33:10.063600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:10.063533 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:33:10.063600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:10.063580 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:33:11.069146 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:11.069116 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" event={"ID":"9eb90821-604b-40ab-8efb-f82ca783417a","Type":"ContainerStarted","Data":"55dbacf4db322b2135b4c87bcefeabfe1a5883f43d5804a491c9c0f5fa6fb1aa"} Jul 21 08:33:11.673192 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:11.673158 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:33:11.674227 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:11.674200 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-dk6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 21 08:33:11.674298 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:11.674257 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" podUID="9eb90821-604b-40ab-8efb-f82ca783417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 21 08:33:12.673542 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:12.673512 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-dk6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 21 08:33:12.673940 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:12.673567 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" podUID="9eb90821-604b-40ab-8efb-f82ca783417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 21 08:33:13.673630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:13.673592 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-dk6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 21 08:33:13.674021 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:13.673649 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" podUID="9eb90821-604b-40ab-8efb-f82ca783417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 21 08:33:14.673216 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:14.673164 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-dk6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body= Jul 21 08:33:14.673417 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:14.673227 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" podUID="9eb90821-604b-40ab-8efb-f82ca783417a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" Jul 21 08:33:15.585119 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:15.585080 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:33:15.677194 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:15.677165 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:33:16.096063 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:16.096038 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-dk6s5" Jul 21 08:33:17.049057 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.049030 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-bmsts" Jul 21 08:33:17.232463 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.232429 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl"] Jul 21 08:33:17.235574 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.235552 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.237938 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.237918 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 08:33:17.238057 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.237993 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 08:33:17.238118 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.238078 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 08:33:17.238172 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.238134 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k878q\"" Jul 21 08:33:17.245668 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.245646 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl"] Jul 21 08:33:17.249525 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.249502 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.249648 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.249529 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.249648 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.249556 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dr8t\" (UniqueName: \"kubernetes.io/projected/bdf15937-3c9b-4526-b821-12fd309b5f36-kube-api-access-6dr8t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.249648 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.249586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.249648 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.249614 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf15937-3c9b-4526-b821-12fd309b5f36-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.249792 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.249661 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.350633 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.350524 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dr8t\" (UniqueName: \"kubernetes.io/projected/bdf15937-3c9b-4526-b821-12fd309b5f36-kube-api-access-6dr8t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.350633 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.350581 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.350633 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.350608 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf15937-3c9b-4526-b821-12fd309b5f36-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.350937 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.350657 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.350937 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.350757 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.350937 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.350782 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.351174 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.351149 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.351239 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.351181 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.351239 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.351216 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.354828 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.354791 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf15937-3c9b-4526-b821-12fd309b5f36-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.355007 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.354878 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf15937-3c9b-4526-b821-12fd309b5f36-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.359260 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.359229 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dr8t\" (UniqueName: \"kubernetes.io/projected/bdf15937-3c9b-4526-b821-12fd309b5f36-kube-api-access-6dr8t\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl\" (UID: \"bdf15937-3c9b-4526-b821-12fd309b5f36\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.550014 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.549969 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:17.696323 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:17.696281 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl"] Jul 21 08:33:17.699076 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:33:17.699049 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf15937_3c9b_4526_b821_12fd309b5f36.slice/crio-876056ae3d66891df538c7a2aa648dcdaf80b5589dc2e48c76c17b1e51660067 WatchSource:0}: Error finding container 876056ae3d66891df538c7a2aa648dcdaf80b5589dc2e48c76c17b1e51660067: Status 404 returned error can't find the container with id 876056ae3d66891df538c7a2aa648dcdaf80b5589dc2e48c76c17b1e51660067 Jul 21 08:33:18.054599 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:18.054446 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-ltznq" Jul 21 08:33:18.111516 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:18.110972 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" event={"ID":"bdf15937-3c9b-4526-b821-12fd309b5f36","Type":"ContainerStarted","Data":"876056ae3d66891df538c7a2aa648dcdaf80b5589dc2e48c76c17b1e51660067"} Jul 21 08:33:19.117523 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:19.117488 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" event={"ID":"bdf15937-3c9b-4526-b821-12fd309b5f36","Type":"ContainerStarted","Data":"818f1f0a14a370d0d33c57f7e758109474c65598a9df18d36b0209f2d203e948"} Jul 21 08:33:19.133461 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:19.133412 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" podStartSLOduration=0.947318522 podStartE2EDuration="2.133393172s" totalImagesPullingTime="1.18607465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:33:19.132723849 +0000 UTC m=+855.018394399" watchObservedRunningTime="2026-07-21 08:33:19.133393172 +0000 UTC m=+855.019063719" Jul 21 08:33:26.341757 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.341724 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn"] Jul 21 08:33:26.344884 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.344862 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.347263 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.347242 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 08:33:26.358100 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.358076 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn"] Jul 21 08:33:26.434456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.434413 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e95a8885-61f4-48ff-8b46-aff8e4de979f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.434610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.434473 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.434610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.434596 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6st\" (UniqueName: \"kubernetes.io/projected/e95a8885-61f4-48ff-8b46-aff8e4de979f-kube-api-access-dq6st\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.434703 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.434631 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.434703 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.434663 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.434770 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.434698 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.535839 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.535785 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e95a8885-61f4-48ff-8b46-aff8e4de979f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.535839 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.535854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.536104 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.535898 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6st\" (UniqueName: \"kubernetes.io/projected/e95a8885-61f4-48ff-8b46-aff8e4de979f-kube-api-access-dq6st\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.536104 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.535918 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.536104 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.535940 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.536104 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.535963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.536487 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.536454 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.536600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.536501 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.536600 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.536528 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.538344 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.538294 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e95a8885-61f4-48ff-8b46-aff8e4de979f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.538928 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.538908 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e95a8885-61f4-48ff-8b46-aff8e4de979f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.543985 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.543956 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6st\" (UniqueName: \"kubernetes.io/projected/e95a8885-61f4-48ff-8b46-aff8e4de979f-kube-api-access-dq6st\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bv6dn\" (UID: \"e95a8885-61f4-48ff-8b46-aff8e4de979f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.655744 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.655649 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:26.798798 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:26.798768 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn"] Jul 21 08:33:26.800897 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:33:26.800864 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95a8885_61f4_48ff_8b46_aff8e4de979f.slice/crio-128dc30bb7b1b0893cb5a273791150cde715f9f5a0b070f1ee103169f0704f3b WatchSource:0}: Error finding container 128dc30bb7b1b0893cb5a273791150cde715f9f5a0b070f1ee103169f0704f3b: Status 404 returned error can't find the container with id 128dc30bb7b1b0893cb5a273791150cde715f9f5a0b070f1ee103169f0704f3b Jul 21 08:33:27.155863 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:27.155833 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" event={"ID":"e95a8885-61f4-48ff-8b46-aff8e4de979f","Type":"ContainerStarted","Data":"128dc30bb7b1b0893cb5a273791150cde715f9f5a0b070f1ee103169f0704f3b"} Jul 21 08:33:27.550579 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:27.550543 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:27.550920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:27.550609 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:27.563328 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:27.563286 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:28.161796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:28.161760 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" event={"ID":"e95a8885-61f4-48ff-8b46-aff8e4de979f","Type":"ContainerStarted","Data":"63b3139786a54d27260df6f7944e8f249659bfe37b5006d45d680f6a9c379f78"} Jul 21 08:33:28.173713 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:28.173688 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl" Jul 21 08:33:28.180609 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:28.180564 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" podStartSLOduration=1.852058219 podStartE2EDuration="2.180548581s" totalImagesPullingTime="328.490362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:33:28.178193825 +0000 UTC m=+864.063864372" watchObservedRunningTime="2026-07-21 08:33:28.180548581 +0000 UTC m=+864.066219123" Jul 21 08:33:36.656109 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:36.656071 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:36.656601 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:36.656166 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:36.674879 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:36.674853 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:33:37.216102 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:33:37.216073 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bv6dn" Jul 21 08:34:04.970442 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:04.970414 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:34:04.971702 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:04.971680 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:34:53.912769 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:53.912732 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6549bdf6bc-l2pfx"] Jul 21 08:34:53.915990 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:53.915967 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:53.926599 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:53.925671 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6549bdf6bc-l2pfx"] Jul 21 08:34:53.966921 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:53.966884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlz8\" (UniqueName: \"kubernetes.io/projected/4d92c825-609a-40dc-8968-f77697b8fda8-kube-api-access-5vlz8\") pod \"authorino-6549bdf6bc-l2pfx\" (UID: \"4d92c825-609a-40dc-8968-f77697b8fda8\") " pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:53.967086 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:53.966926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d92c825-609a-40dc-8968-f77697b8fda8-tls-cert\") pod \"authorino-6549bdf6bc-l2pfx\" (UID: \"4d92c825-609a-40dc-8968-f77697b8fda8\") " pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:54.068405 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:54.068365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlz8\" (UniqueName: \"kubernetes.io/projected/4d92c825-609a-40dc-8968-f77697b8fda8-kube-api-access-5vlz8\") pod \"authorino-6549bdf6bc-l2pfx\" (UID: \"4d92c825-609a-40dc-8968-f77697b8fda8\") " pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:54.068405 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:54.068406 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d92c825-609a-40dc-8968-f77697b8fda8-tls-cert\") pod \"authorino-6549bdf6bc-l2pfx\" (UID: \"4d92c825-609a-40dc-8968-f77697b8fda8\") " pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:54.071169 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:54.071142 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d92c825-609a-40dc-8968-f77697b8fda8-tls-cert\") pod \"authorino-6549bdf6bc-l2pfx\" (UID: \"4d92c825-609a-40dc-8968-f77697b8fda8\") " pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:54.075970 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:54.075950 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlz8\" (UniqueName: \"kubernetes.io/projected/4d92c825-609a-40dc-8968-f77697b8fda8-kube-api-access-5vlz8\") pod \"authorino-6549bdf6bc-l2pfx\" (UID: \"4d92c825-609a-40dc-8968-f77697b8fda8\") " pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:54.228108 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:54.228071 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" Jul 21 08:34:54.360200 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:54.360174 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6549bdf6bc-l2pfx"] Jul 21 08:34:54.361952 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:34:54.361918 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d92c825_609a_40dc_8968_f77697b8fda8.slice/crio-0dc6644a2e7d648879fd59aa2063a4b5922b82b18fd37d5c9d0143711e289d12 WatchSource:0}: Error finding container 0dc6644a2e7d648879fd59aa2063a4b5922b82b18fd37d5c9d0143711e289d12: Status 404 returned error can't find the container with id 0dc6644a2e7d648879fd59aa2063a4b5922b82b18fd37d5c9d0143711e289d12 Jul 21 08:34:54.592596 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:54.592511 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" event={"ID":"4d92c825-609a-40dc-8968-f77697b8fda8","Type":"ContainerStarted","Data":"0dc6644a2e7d648879fd59aa2063a4b5922b82b18fd37d5c9d0143711e289d12"} Jul 21 08:34:55.599691 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:55.599581 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" event={"ID":"4d92c825-609a-40dc-8968-f77697b8fda8","Type":"ContainerStarted","Data":"2b06649b5f664a3980a54122ce0147ee7be3d4ab1a398bda0ca4de665acea6e8"} Jul 21 08:34:55.624093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:55.624029 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6549bdf6bc-l2pfx" podStartSLOduration=2.198948288 podStartE2EDuration="2.624007627s" totalImagesPullingTime="425.059339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:34:55.62298727 +0000 UTC m=+951.508657830" watchObservedRunningTime="2026-07-21 08:34:55.624007627 +0000 UTC m=+951.509678180" Jul 21 08:34:55.682519 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:55.682474 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-799d4bf97d-8x8q5"] Jul 21 08:34:55.683255 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:55.682723 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" podUID="8b48673b-b50e-4a28-94ec-c130b7dec4b7" containerName="authorino" containerID="cri-o://15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0" gracePeriod=30 Jul 21 08:34:55.953106 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:55.953066 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:34:56.087619 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.087583 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b48673b-b50e-4a28-94ec-c130b7dec4b7-tls-cert\") pod \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " Jul 21 08:34:56.087619 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.087623 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbmrc\" (UniqueName: \"kubernetes.io/projected/8b48673b-b50e-4a28-94ec-c130b7dec4b7-kube-api-access-bbmrc\") pod \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\" (UID: \"8b48673b-b50e-4a28-94ec-c130b7dec4b7\") " Jul 21 08:34:56.089824 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.089780 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b48673b-b50e-4a28-94ec-c130b7dec4b7-kube-api-access-bbmrc" (OuterVolumeSpecName: "kube-api-access-bbmrc") pod "8b48673b-b50e-4a28-94ec-c130b7dec4b7" (UID: "8b48673b-b50e-4a28-94ec-c130b7dec4b7"). InnerVolumeSpecName "kube-api-access-bbmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:34:56.098555 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.098525 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b48673b-b50e-4a28-94ec-c130b7dec4b7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8b48673b-b50e-4a28-94ec-c130b7dec4b7" (UID: "8b48673b-b50e-4a28-94ec-c130b7dec4b7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:34:56.189462 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.189428 2553 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8b48673b-b50e-4a28-94ec-c130b7dec4b7-tls-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:34:56.189462 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.189462 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmrc\" (UniqueName: \"kubernetes.io/projected/8b48673b-b50e-4a28-94ec-c130b7dec4b7-kube-api-access-bbmrc\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:34:56.606818 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.606783 2553 generic.go:356] "Generic (PLEG): container finished" podID="8b48673b-b50e-4a28-94ec-c130b7dec4b7" containerID="15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0" exitCode=0 Jul 21 08:34:56.607342 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.606828 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" Jul 21 08:34:56.607342 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.606860 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" event={"ID":"8b48673b-b50e-4a28-94ec-c130b7dec4b7","Type":"ContainerDied","Data":"15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0"} Jul 21 08:34:56.607342 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.606906 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-799d4bf97d-8x8q5" event={"ID":"8b48673b-b50e-4a28-94ec-c130b7dec4b7","Type":"ContainerDied","Data":"55e6086a3580d388d53437d43130f6dcae981a17fac62313bcf536c5000c437d"} Jul 21 08:34:56.607342 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.606927 2553 scope.go:117] "RemoveContainer" containerID="15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0" Jul 21 08:34:56.617546 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.617512 2553 scope.go:117] "RemoveContainer" containerID="15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0" Jul 21 08:34:56.617946 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:34:56.617924 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0\": container with ID starting with 15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0 not found: ID does not exist" containerID="15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0" Jul 21 08:34:56.618054 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.617951 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0"} err="failed to get container status \"15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0\": rpc error: code = NotFound desc = could not find container \"15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0\": container with ID starting with 15be8533323e77c1cfc3246cb01089134d114b001f670ca987cc794f4e2477b0 not found: ID does not exist" Jul 21 08:34:56.631861 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.631837 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-799d4bf97d-8x8q5"] Jul 21 08:34:56.636073 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.636049 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-799d4bf97d-8x8q5"] Jul 21 08:34:56.674439 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:34:56.674399 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b48673b-b50e-4a28-94ec-c130b7dec4b7" path="/var/lib/kubelet/pods/8b48673b-b50e-4a28-94ec-c130b7dec4b7/volumes" Jul 21 08:35:53.677718 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:53.677678 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4hjl9"] Jul 21 08:35:53.678284 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:53.677891 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" podUID="f38df518-35a9-461b-a365-96f3bb967902" containerName="manager" containerID="cri-o://fc2160665d92720f670dbfed8be6b4909cd233f4c6306c89dc91eccc00b56aea" gracePeriod=30 Jul 21 08:35:53.894884 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:53.894851 2553 generic.go:356] "Generic (PLEG): container finished" podID="f38df518-35a9-461b-a365-96f3bb967902" containerID="fc2160665d92720f670dbfed8be6b4909cd233f4c6306c89dc91eccc00b56aea" exitCode=0 Jul 21 08:35:53.895048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:53.894928 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" event={"ID":"f38df518-35a9-461b-a365-96f3bb967902","Type":"ContainerDied","Data":"fc2160665d92720f670dbfed8be6b4909cd233f4c6306c89dc91eccc00b56aea"} Jul 21 08:35:53.935493 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:53.935431 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:35:54.034205 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.034160 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btxz9\" (UniqueName: \"kubernetes.io/projected/f38df518-35a9-461b-a365-96f3bb967902-kube-api-access-btxz9\") pod \"f38df518-35a9-461b-a365-96f3bb967902\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " Jul 21 08:35:54.034440 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.034256 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f38df518-35a9-461b-a365-96f3bb967902-webhook-cert\") pod \"f38df518-35a9-461b-a365-96f3bb967902\" (UID: \"f38df518-35a9-461b-a365-96f3bb967902\") " Jul 21 08:35:54.036348 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.036321 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38df518-35a9-461b-a365-96f3bb967902-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f38df518-35a9-461b-a365-96f3bb967902" (UID: "f38df518-35a9-461b-a365-96f3bb967902"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:35:54.036439 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.036348 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38df518-35a9-461b-a365-96f3bb967902-kube-api-access-btxz9" (OuterVolumeSpecName: "kube-api-access-btxz9") pod "f38df518-35a9-461b-a365-96f3bb967902" (UID: "f38df518-35a9-461b-a365-96f3bb967902"). InnerVolumeSpecName "kube-api-access-btxz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:35:54.135332 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.135285 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btxz9\" (UniqueName: \"kubernetes.io/projected/f38df518-35a9-461b-a365-96f3bb967902-kube-api-access-btxz9\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:35:54.135507 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.135361 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f38df518-35a9-461b-a365-96f3bb967902-webhook-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:35:54.901093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.901056 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" Jul 21 08:35:54.901093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.901078 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-4hjl9" event={"ID":"f38df518-35a9-461b-a365-96f3bb967902","Type":"ContainerDied","Data":"0d654fcac293240874fc9e4cf56b70ecf2e06e4caf5cab6cf81ffe336cef53c7"} Jul 21 08:35:54.901592 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.901147 2553 scope.go:117] "RemoveContainer" containerID="fc2160665d92720f670dbfed8be6b4909cd233f4c6306c89dc91eccc00b56aea" Jul 21 08:35:54.918076 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.918047 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4hjl9"] Jul 21 08:35:54.921887 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:54.921865 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4hjl9"] Jul 21 08:35:56.675020 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:35:56.674989 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38df518-35a9-461b-a365-96f3bb967902" path="/var/lib/kubelet/pods/f38df518-35a9-461b-a365-96f3bb967902/volumes" Jul 21 08:36:55.740821 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.740742 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d799f9c8-n2nlf"] Jul 21 08:36:55.741470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.741136 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38df518-35a9-461b-a365-96f3bb967902" containerName="manager" Jul 21 08:36:55.741470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.741148 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38df518-35a9-461b-a365-96f3bb967902" containerName="manager" Jul 21 08:36:55.741470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.741167 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b48673b-b50e-4a28-94ec-c130b7dec4b7" containerName="authorino" Jul 21 08:36:55.741470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.741173 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b48673b-b50e-4a28-94ec-c130b7dec4b7" containerName="authorino" Jul 21 08:36:55.741470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.741234 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38df518-35a9-461b-a365-96f3bb967902" containerName="manager" Jul 21 08:36:55.741470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.741244 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b48673b-b50e-4a28-94ec-c130b7dec4b7" containerName="authorino" Jul 21 08:36:55.744331 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.744299 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:55.746966 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.746941 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 08:36:55.747100 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.746987 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 08:36:55.747689 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.747674 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zmxb6\"" Jul 21 08:36:55.753290 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.753255 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-n2nlf"] Jul 21 08:36:55.800451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.800418 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f655e372-88f8-4639-b7fe-4c461d057a44-webhook-cert\") pod \"maas-controller-68d799f9c8-n2nlf\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:55.800641 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.800572 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxjs\" (UniqueName: \"kubernetes.io/projected/f655e372-88f8-4639-b7fe-4c461d057a44-kube-api-access-frxjs\") pod \"maas-controller-68d799f9c8-n2nlf\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:55.901370 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.901292 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxjs\" (UniqueName: \"kubernetes.io/projected/f655e372-88f8-4639-b7fe-4c461d057a44-kube-api-access-frxjs\") pod \"maas-controller-68d799f9c8-n2nlf\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:55.901546 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.901413 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f655e372-88f8-4639-b7fe-4c461d057a44-webhook-cert\") pod \"maas-controller-68d799f9c8-n2nlf\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:55.903961 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.903931 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f655e372-88f8-4639-b7fe-4c461d057a44-webhook-cert\") pod \"maas-controller-68d799f9c8-n2nlf\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:55.911197 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:55.911170 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxjs\" (UniqueName: \"kubernetes.io/projected/f655e372-88f8-4639-b7fe-4c461d057a44-kube-api-access-frxjs\") pod \"maas-controller-68d799f9c8-n2nlf\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:56.056508 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:56.056417 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:56.196271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:56.196246 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-n2nlf"] Jul 21 08:36:56.198250 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:36:56.198222 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf655e372_88f8_4639_b7fe_4c461d057a44.slice/crio-c656deae9fdee3398a5119a54e508bcfa6676efeeacd49f3a48d0c7daa540e2e WatchSource:0}: Error finding container c656deae9fdee3398a5119a54e508bcfa6676efeeacd49f3a48d0c7daa540e2e: Status 404 returned error can't find the container with id c656deae9fdee3398a5119a54e508bcfa6676efeeacd49f3a48d0c7daa540e2e Jul 21 08:36:56.199534 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:56.199518 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:36:57.204670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:57.204628 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" event={"ID":"f655e372-88f8-4639-b7fe-4c461d057a44","Type":"ContainerStarted","Data":"f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d"} Jul 21 08:36:57.205031 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:57.204692 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" event={"ID":"f655e372-88f8-4639-b7fe-4c461d057a44","Type":"ContainerStarted","Data":"c656deae9fdee3398a5119a54e508bcfa6676efeeacd49f3a48d0c7daa540e2e"} Jul 21 08:36:57.205031 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:57.204748 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:36:57.223435 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:36:57.223380 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" podStartSLOduration=1.767437829 podStartE2EDuration="2.223363159s" totalImagesPullingTime="455.92533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:36:57.223187375 +0000 UTC m=+1073.108857946" watchObservedRunningTime="2026-07-21 08:36:57.223363159 +0000 UTC m=+1073.109033708" Jul 21 08:37:08.212249 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:08.212217 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:37:12.872571 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:12.872537 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-n2nlf"] Jul 21 08:37:12.872998 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:12.872715 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" podUID="f655e372-88f8-4639-b7fe-4c461d057a44" containerName="manager" containerID="cri-o://f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d" gracePeriod=30 Jul 21 08:37:13.125767 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.125708 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:37:13.261034 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.261002 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f655e372-88f8-4639-b7fe-4c461d057a44-webhook-cert\") pod \"f655e372-88f8-4639-b7fe-4c461d057a44\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " Jul 21 08:37:13.261217 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.261093 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frxjs\" (UniqueName: \"kubernetes.io/projected/f655e372-88f8-4639-b7fe-4c461d057a44-kube-api-access-frxjs\") pod \"f655e372-88f8-4639-b7fe-4c461d057a44\" (UID: \"f655e372-88f8-4639-b7fe-4c461d057a44\") " Jul 21 08:37:13.263239 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.263203 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f655e372-88f8-4639-b7fe-4c461d057a44-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f655e372-88f8-4639-b7fe-4c461d057a44" (UID: "f655e372-88f8-4639-b7fe-4c461d057a44"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:37:13.263375 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.263244 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f655e372-88f8-4639-b7fe-4c461d057a44-kube-api-access-frxjs" (OuterVolumeSpecName: "kube-api-access-frxjs") pod "f655e372-88f8-4639-b7fe-4c461d057a44" (UID: "f655e372-88f8-4639-b7fe-4c461d057a44"). InnerVolumeSpecName "kube-api-access-frxjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:37:13.290402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.290366 2553 generic.go:356] "Generic (PLEG): container finished" podID="f655e372-88f8-4639-b7fe-4c461d057a44" containerID="f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d" exitCode=0 Jul 21 08:37:13.290567 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.290415 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" event={"ID":"f655e372-88f8-4639-b7fe-4c461d057a44","Type":"ContainerDied","Data":"f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d"} Jul 21 08:37:13.290567 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.290420 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" Jul 21 08:37:13.290567 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.290479 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-n2nlf" event={"ID":"f655e372-88f8-4639-b7fe-4c461d057a44","Type":"ContainerDied","Data":"c656deae9fdee3398a5119a54e508bcfa6676efeeacd49f3a48d0c7daa540e2e"} Jul 21 08:37:13.290567 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.290501 2553 scope.go:117] "RemoveContainer" containerID="f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d" Jul 21 08:37:13.299544 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.299523 2553 scope.go:117] "RemoveContainer" containerID="f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d" Jul 21 08:37:13.299799 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:37:13.299779 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d\": container with ID starting with f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d not found: ID does not exist" containerID="f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d" Jul 21 08:37:13.299870 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.299811 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d"} err="failed to get container status \"f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d\": rpc error: code = NotFound desc = could not find container \"f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d\": container with ID starting with f6e8fb61ab67d6b56474fd5a24bf52358768a5b0e281a267bbe20513e023bd6d not found: ID does not exist" Jul 21 08:37:13.312011 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.311982 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-n2nlf"] Jul 21 08:37:13.318822 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.318797 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-n2nlf"] Jul 21 08:37:13.362065 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.362033 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxjs\" (UniqueName: \"kubernetes.io/projected/f655e372-88f8-4639-b7fe-4c461d057a44-kube-api-access-frxjs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:37:13.362065 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:13.362060 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f655e372-88f8-4639-b7fe-4c461d057a44-webhook-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:37:14.186861 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.186832 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4glzb"] Jul 21 08:37:14.187250 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.187237 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f655e372-88f8-4639-b7fe-4c461d057a44" containerName="manager" Jul 21 08:37:14.187250 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.187251 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f655e372-88f8-4639-b7fe-4c461d057a44" containerName="manager" Jul 21 08:37:14.187345 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.187335 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f655e372-88f8-4639-b7fe-4c461d057a44" containerName="manager" Jul 21 08:37:14.191845 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.191820 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.195449 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.195426 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zmxb6\"" Jul 21 08:37:14.195589 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.195449 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 08:37:14.195589 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.195470 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 08:37:14.200911 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.200889 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4glzb"] Jul 21 08:37:14.271171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.271131 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct8zg\" (UniqueName: \"kubernetes.io/projected/7f76cd9b-efd8-4e60-8e76-f87f779db044-kube-api-access-ct8zg\") pod \"maas-controller-68d799f9c8-4glzb\" (UID: \"7f76cd9b-efd8-4e60-8e76-f87f779db044\") " pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.271392 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.271216 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f76cd9b-efd8-4e60-8e76-f87f779db044-webhook-cert\") pod \"maas-controller-68d799f9c8-4glzb\" (UID: \"7f76cd9b-efd8-4e60-8e76-f87f779db044\") " pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.372432 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.372392 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct8zg\" (UniqueName: \"kubernetes.io/projected/7f76cd9b-efd8-4e60-8e76-f87f779db044-kube-api-access-ct8zg\") pod \"maas-controller-68d799f9c8-4glzb\" (UID: \"7f76cd9b-efd8-4e60-8e76-f87f779db044\") " pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.372612 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.372462 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f76cd9b-efd8-4e60-8e76-f87f779db044-webhook-cert\") pod \"maas-controller-68d799f9c8-4glzb\" (UID: \"7f76cd9b-efd8-4e60-8e76-f87f779db044\") " pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.374986 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.374955 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f76cd9b-efd8-4e60-8e76-f87f779db044-webhook-cert\") pod \"maas-controller-68d799f9c8-4glzb\" (UID: \"7f76cd9b-efd8-4e60-8e76-f87f779db044\") " pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.381464 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.381438 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct8zg\" (UniqueName: \"kubernetes.io/projected/7f76cd9b-efd8-4e60-8e76-f87f779db044-kube-api-access-ct8zg\") pod \"maas-controller-68d799f9c8-4glzb\" (UID: \"7f76cd9b-efd8-4e60-8e76-f87f779db044\") " pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.503814 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.503726 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:14.653013 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.652976 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d799f9c8-4glzb"] Jul 21 08:37:14.655095 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:37:14.655068 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f76cd9b_efd8_4e60_8e76_f87f779db044.slice/crio-8e0b100631d8cae55f6448cc4aff67379204bb20cf3ad0bbb513066ce58da8c2 WatchSource:0}: Error finding container 8e0b100631d8cae55f6448cc4aff67379204bb20cf3ad0bbb513066ce58da8c2: Status 404 returned error can't find the container with id 8e0b100631d8cae55f6448cc4aff67379204bb20cf3ad0bbb513066ce58da8c2 Jul 21 08:37:14.677543 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:14.677506 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f655e372-88f8-4639-b7fe-4c461d057a44" path="/var/lib/kubelet/pods/f655e372-88f8-4639-b7fe-4c461d057a44/volumes" Jul 21 08:37:15.302906 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:15.302866 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-4glzb" event={"ID":"7f76cd9b-efd8-4e60-8e76-f87f779db044","Type":"ContainerStarted","Data":"65e2b296782b5e3f1b61ba17524616301e7158b41a17fb55c1ccfee4d2f36000"} Jul 21 08:37:15.303262 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:15.302929 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d799f9c8-4glzb" event={"ID":"7f76cd9b-efd8-4e60-8e76-f87f779db044","Type":"ContainerStarted","Data":"8e0b100631d8cae55f6448cc4aff67379204bb20cf3ad0bbb513066ce58da8c2"} Jul 21 08:37:15.303262 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:15.302952 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:37:15.330212 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:15.330160 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d799f9c8-4glzb" podStartSLOduration=0.881535971 podStartE2EDuration="1.33014074s" totalImagesPullingTime="448.604769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:37:15.327476115 +0000 UTC m=+1091.213146683" watchObservedRunningTime="2026-07-21 08:37:15.33014074 +0000 UTC m=+1091.215811288" Jul 21 08:37:26.309740 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:37:26.309710 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d799f9c8-4glzb" Jul 21 08:39:05.007289 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:39:05.007260 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:39:05.009417 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:39:05.009397 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:44:05.037895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:44:05.037863 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:44:05.041558 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:44:05.041536 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:49:05.073737 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:05.073710 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:49:05.080267 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:05.080244 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:49:48.212869 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.212832 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5"] Jul 21 08:49:48.216647 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.216624 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.219589 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.219567 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-68712b6b-openshift-default-dockercfg-tpdjz\"" Jul 21 08:49:48.236954 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.236927 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5"] Jul 21 08:49:48.328093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328059 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-podinfo\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328094 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-token\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328302 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328129 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-socket\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328302 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328216 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-certs\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328302 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328272 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-credential-socket\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328302 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328289 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-envoy\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328490 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328363 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-data\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328490 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328391 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istiod-ca-cert\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.328490 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.328411 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrb7d\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-kube-api-access-wrb7d\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429429 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429389 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-credential-socket\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429429 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429432 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-envoy\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429686 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429479 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-data\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429686 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429514 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istiod-ca-cert\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429686 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrb7d\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-kube-api-access-wrb7d\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429686 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429606 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-podinfo\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429686 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429627 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-token\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429686 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-socket\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429713 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-certs\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429844 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-credential-socket\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.429993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.429884 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-data\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.430281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.430262 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-certs\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.430346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.430291 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-socket\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.430394 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.430334 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istiod-ca-cert\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.431960 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.431936 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-envoy\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.432421 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.432398 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-podinfo\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.437541 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.437520 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-token\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.437697 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.437680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrb7d\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-kube-api-access-wrb7d\") pod \"e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.527080 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.526992 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:48.674788 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.674764 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5"] Jul 21 08:49:48.676534 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:49:48.676500 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f9a82a_77d8_46e0_8175_b6e8b65232ad.slice/crio-aca3fcfc4345ded34cf0dcd0968bb5d953a97ee12c30d420bdf02597f0eab648 WatchSource:0}: Error finding container aca3fcfc4345ded34cf0dcd0968bb5d953a97ee12c30d420bdf02597f0eab648: Status 404 returned error can't find the container with id aca3fcfc4345ded34cf0dcd0968bb5d953a97ee12c30d420bdf02597f0eab648 Jul 21 08:49:48.678805 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.678766 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:49:48.678895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.678849 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:49:48.678895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:48.678879 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:49:49.080360 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:49.074592 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" event={"ID":"15f9a82a-77d8-46e0-8175-b6e8b65232ad","Type":"ContainerStarted","Data":"7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4"} Jul 21 08:49:49.080360 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:49.074734 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" event={"ID":"15f9a82a-77d8-46e0-8175-b6e8b65232ad","Type":"ContainerStarted","Data":"aca3fcfc4345ded34cf0dcd0968bb5d953a97ee12c30d420bdf02597f0eab648"} Jul 21 08:49:49.099441 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:49.099382 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" podStartSLOduration=1.099364703 podStartE2EDuration="1.099364703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:49:49.097506343 +0000 UTC m=+1844.983176892" watchObservedRunningTime="2026-07-21 08:49:49.099364703 +0000 UTC m=+1844.985035251" Jul 21 08:49:49.528072 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:49.528032 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:49.533180 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:49.533156 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:50.080347 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.080300 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:50.081437 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.081406 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:49:50.573596 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.573561 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm"] Jul 21 08:49:50.577240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.577218 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.579809 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.579777 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-68712b6b-serving-cert\"" Jul 21 08:49:50.591606 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.591580 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm"] Jul 21 08:49:50.649996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.649956 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-maas-api-tls\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.650167 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.650029 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vpls\" (UniqueName: \"kubernetes.io/projected/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-kube-api-access-8vpls\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.650167 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.650082 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-openshift-service-ca\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.751072 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.751037 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vpls\" (UniqueName: \"kubernetes.io/projected/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-kube-api-access-8vpls\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.751072 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.751073 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-openshift-service-ca\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.751433 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.751137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-maas-api-tls\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.751880 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.751860 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-openshift-service-ca\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.753755 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.753730 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-maas-api-tls\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.760060 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.760035 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vpls\" (UniqueName: \"kubernetes.io/projected/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-kube-api-access-8vpls\") pod \"maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:50.888682 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:50.888600 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:51.036335 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:51.036279 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm"] Jul 21 08:49:51.037178 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:49:51.037148 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf9c632_b72c_45ac_bee3_eae2ef05ca21.slice/crio-f641b3c604fdad290d63f231336331a86463e3f7004f26696342c7af8530de06 WatchSource:0}: Error finding container f641b3c604fdad290d63f231336331a86463e3f7004f26696342c7af8530de06: Status 404 returned error can't find the container with id f641b3c604fdad290d63f231336331a86463e3f7004f26696342c7af8530de06 Jul 21 08:49:51.038732 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:51.038710 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:49:51.087451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:51.087416 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" event={"ID":"1bf9c632-b72c-45ac-bee3-eae2ef05ca21","Type":"ContainerStarted","Data":"f641b3c604fdad290d63f231336331a86463e3f7004f26696342c7af8530de06"} Jul 21 08:49:52.094624 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:52.094581 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" event={"ID":"1bf9c632-b72c-45ac-bee3-eae2ef05ca21","Type":"ContainerStarted","Data":"de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab"} Jul 21 08:49:52.132614 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:52.132560 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" podStartSLOduration=1.581019184 podStartE2EDuration="2.132541613s" totalImagesPullingTime="551.522429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:49:52.131573728 +0000 UTC m=+1848.017244276" watchObservedRunningTime="2026-07-21 08:49:52.132541613 +0000 UTC m=+1848.018212160" Jul 21 08:49:53.099660 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:53.099631 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:49:59.113037 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:49:59.113000 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:50:06.121647 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:06.121584 2553 patch_prober.go:28] interesting pod/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 08:50:06.122044 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:06.121682 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 08:50:07.173839 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:07.173804 2553 generic.go:356] "Generic (PLEG): container finished" podID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerID="7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4" exitCode=0 Jul 21 08:50:07.174281 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:07.173880 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" event={"ID":"15f9a82a-77d8-46e0-8175-b6e8b65232ad","Type":"ContainerDied","Data":"7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4"} Jul 21 08:50:07.174358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:07.174323 2553 scope.go:117] "RemoveContainer" containerID="7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4" Jul 21 08:50:07.175386 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:07.175357 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:07.175462 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:07.175447 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:07.175499 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:07.175488 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:08.180750 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:08.180715 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" event={"ID":"15f9a82a-77d8-46e0-8175-b6e8b65232ad","Type":"ContainerStarted","Data":"ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648"} Jul 21 08:50:08.527435 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:08.527346 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:50:08.527596 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:08.527574 2553 patch_prober.go:28] interesting pod/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 08:50:08.527639 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:08.527620 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 08:50:09.527331 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:09.527279 2553 patch_prober.go:28] interesting pod/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 08:50:09.527856 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:09.527348 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 08:50:10.528214 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:10.528173 2553 patch_prober.go:28] interesting pod/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 08:50:10.528649 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:10.528243 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 08:50:11.527745 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:11.527710 2553 patch_prober.go:28] interesting pod/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 21 08:50:11.527929 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:11.527770 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 21 08:50:12.531493 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:12.531464 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:50:12.531864 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:12.531799 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:50:12.532521 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:12.532501 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:50:26.619892 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:26.619806 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm"] Jul 21 08:50:26.620442 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:26.620074 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" podUID="1bf9c632-b72c-45ac-bee3-eae2ef05ca21" containerName="maas-api" containerID="cri-o://de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab" gracePeriod=30 Jul 21 08:50:26.878375 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:26.878294 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:50:27.009095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.009062 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vpls\" (UniqueName: \"kubernetes.io/projected/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-kube-api-access-8vpls\") pod \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " Jul 21 08:50:27.009276 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.009142 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-maas-api-tls\") pod \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " Jul 21 08:50:27.009276 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.009174 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-openshift-service-ca\") pod \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\" (UID: \"1bf9c632-b72c-45ac-bee3-eae2ef05ca21\") " Jul 21 08:50:27.009648 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.009612 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1bf9c632-b72c-45ac-bee3-eae2ef05ca21" (UID: "1bf9c632-b72c-45ac-bee3-eae2ef05ca21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:50:27.011616 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.011591 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1bf9c632-b72c-45ac-bee3-eae2ef05ca21" (UID: "1bf9c632-b72c-45ac-bee3-eae2ef05ca21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:50:27.011810 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.011788 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-kube-api-access-8vpls" (OuterVolumeSpecName: "kube-api-access-8vpls") pod "1bf9c632-b72c-45ac-bee3-eae2ef05ca21" (UID: "1bf9c632-b72c-45ac-bee3-eae2ef05ca21"). InnerVolumeSpecName "kube-api-access-8vpls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:50:27.110168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.110122 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vpls\" (UniqueName: \"kubernetes.io/projected/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-kube-api-access-8vpls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:27.110168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.110156 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:27.110168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.110175 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf9c632-b72c-45ac-bee3-eae2ef05ca21-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:27.282259 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.282220 2553 generic.go:356] "Generic (PLEG): container finished" podID="1bf9c632-b72c-45ac-bee3-eae2ef05ca21" containerID="de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab" exitCode=0 Jul 21 08:50:27.282499 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.282267 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" Jul 21 08:50:27.282499 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.282301 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" event={"ID":"1bf9c632-b72c-45ac-bee3-eae2ef05ca21","Type":"ContainerDied","Data":"de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab"} Jul 21 08:50:27.282499 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.282383 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm" event={"ID":"1bf9c632-b72c-45ac-bee3-eae2ef05ca21","Type":"ContainerDied","Data":"f641b3c604fdad290d63f231336331a86463e3f7004f26696342c7af8530de06"} Jul 21 08:50:27.282499 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.282412 2553 scope.go:117] "RemoveContainer" containerID="de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab" Jul 21 08:50:27.292732 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.292706 2553 scope.go:117] "RemoveContainer" containerID="de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab" Jul 21 08:50:27.293026 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:50:27.293005 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab\": container with ID starting with de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab not found: ID does not exist" containerID="de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab" Jul 21 08:50:27.293079 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.293038 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab"} err="failed to get container status \"de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab\": rpc error: code = NotFound desc = could not find container \"de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab\": container with ID starting with de685b660a9842514c2f7d2a9f70b6b363aaa230c3a509a4806d14cabb4b7fab not found: ID does not exist" Jul 21 08:50:27.305350 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.305296 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm"] Jul 21 08:50:27.309126 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:27.309102 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68712b6b-5f87fc899c-j2qqm"] Jul 21 08:50:28.674535 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:28.674493 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf9c632-b72c-45ac-bee3-eae2ef05ca21" path="/var/lib/kubelet/pods/1bf9c632-b72c-45ac-bee3-eae2ef05ca21/volumes" Jul 21 08:50:31.283876 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:31.283832 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5"] Jul 21 08:50:31.284382 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:31.284089 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" containerID="cri-o://ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648" gracePeriod=30 Jul 21 08:50:32.537813 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.537779 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2"] Jul 21 08:50:32.538221 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.538198 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf9c632-b72c-45ac-bee3-eae2ef05ca21" containerName="maas-api" Jul 21 08:50:32.538221 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.538210 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf9c632-b72c-45ac-bee3-eae2ef05ca21" containerName="maas-api" Jul 21 08:50:32.538323 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.538300 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf9c632-b72c-45ac-bee3-eae2ef05ca21" containerName="maas-api" Jul 21 08:50:32.541620 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.541599 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.548577 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.548551 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-500e3541-gw-openshift-default-dockercfg-hbkwt\"" Jul 21 08:50:32.568876 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.568845 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2"] Jul 21 08:50:32.661912 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.661874 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-envoy\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.661912 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.661912 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-credential-socket\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.662131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.661938 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-data\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.662131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.661957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee641070-3cb0-4047-a291-09bac166fe2f-istio-podinfo\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.662131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.662035 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee641070-3cb0-4047-a291-09bac166fe2f-istiod-ca-cert\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.662131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.662073 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-socket\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.662131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.662109 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nb6z\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-kube-api-access-9nb6z\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.662131 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.662131 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-certs\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.662371 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.662179 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-istio-token\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.762922 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.762879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee641070-3cb0-4047-a291-09bac166fe2f-istiod-ca-cert\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763125 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.762932 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-socket\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763125 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.762965 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nb6z\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-kube-api-access-9nb6z\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763125 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.762991 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-certs\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763125 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763035 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-istio-token\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763385 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763135 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-envoy\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763385 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763158 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-credential-socket\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763385 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-data\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763385 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763239 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee641070-3cb0-4047-a291-09bac166fe2f-istio-podinfo\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763594 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763423 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-socket\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763594 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763477 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-certs\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763705 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763644 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-data\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763763 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763698 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-credential-socket\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.763763 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.763728 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee641070-3cb0-4047-a291-09bac166fe2f-istiod-ca-cert\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.765671 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.765638 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-envoy\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.765915 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.765899 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee641070-3cb0-4047-a291-09bac166fe2f-istio-podinfo\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.774755 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.774731 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nb6z\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-kube-api-access-9nb6z\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.776069 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.776052 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-istio-token\") pod \"e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:32.853193 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:32.853104 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:33.007115 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.007085 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2"] Jul 21 08:50:33.009606 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:50:33.009577 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee641070_3cb0_4047_a291_09bac166fe2f.slice/crio-6984af358a1a45f6f658d43d71622edc115ede2f114d4439aed1936f11b0793f WatchSource:0}: Error finding container 6984af358a1a45f6f658d43d71622edc115ede2f114d4439aed1936f11b0793f: Status 404 returned error can't find the container with id 6984af358a1a45f6f658d43d71622edc115ede2f114d4439aed1936f11b0793f Jul 21 08:50:33.011691 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.011658 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:33.011760 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.011745 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:33.011803 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.011775 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:33.314945 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.314905 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" event={"ID":"ee641070-3cb0-4047-a291-09bac166fe2f","Type":"ContainerStarted","Data":"a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39"} Jul 21 08:50:33.315159 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.314975 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" event={"ID":"ee641070-3cb0-4047-a291-09bac166fe2f","Type":"ContainerStarted","Data":"6984af358a1a45f6f658d43d71622edc115ede2f114d4439aed1936f11b0793f"} Jul 21 08:50:33.340351 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.340265 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" podStartSLOduration=1.340249579 podStartE2EDuration="1.340249579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:50:33.338056318 +0000 UTC m=+1889.223726876" watchObservedRunningTime="2026-07-21 08:50:33.340249579 +0000 UTC m=+1889.225920127" Jul 21 08:50:33.854272 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:33.854237 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:34.859092 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:34.859052 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:35.325816 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:35.325780 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:35.326797 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:35.326780 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:36.548610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.548545 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:50:36.702234 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702199 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-certs\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702234 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702238 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrb7d\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-kube-api-access-wrb7d\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702503 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702255 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-credential-socket\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702503 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702295 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-token\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702503 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702368 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-data\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702503 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702406 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-envoy\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702709 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702503 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-podinfo\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702709 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702551 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istiod-ca-cert\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702709 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702567 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:50:36.702709 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702603 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-socket\") pod \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\" (UID: \"15f9a82a-77d8-46e0-8175-b6e8b65232ad\") " Jul 21 08:50:36.702709 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702643 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:50:36.703010 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.702986 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.703150 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.703115 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.703150 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.703006 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:50:36.703270 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.703054 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:50:36.703628 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.703601 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:50:36.704808 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.704785 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-kube-api-access-wrb7d" (OuterVolumeSpecName: "kube-api-access-wrb7d") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "kube-api-access-wrb7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:50:36.705006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.704993 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:50:36.705185 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.705170 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 08:50:36.705233 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.705194 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15f9a82a-77d8-46e0-8175-b6e8b65232ad" (UID: "15f9a82a-77d8-46e0-8175-b6e8b65232ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:50:36.804451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.804357 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.804451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.804395 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.804451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.804405 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.804451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.804414 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrb7d\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-kube-api-access-wrb7d\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.804451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.804423 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.804451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.804434 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:36.804451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:36.804442 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f9a82a-77d8-46e0-8175-b6e8b65232ad-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:50:37.338006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.337972 2553 generic.go:356] "Generic (PLEG): container finished" podID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerID="ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648" exitCode=0 Jul 21 08:50:37.338196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.338024 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" Jul 21 08:50:37.338196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.338064 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" event={"ID":"15f9a82a-77d8-46e0-8175-b6e8b65232ad","Type":"ContainerDied","Data":"ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648"} Jul 21 08:50:37.338196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.338118 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5" event={"ID":"15f9a82a-77d8-46e0-8175-b6e8b65232ad","Type":"ContainerDied","Data":"aca3fcfc4345ded34cf0dcd0968bb5d953a97ee12c30d420bdf02597f0eab648"} Jul 21 08:50:37.338196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.338142 2553 scope.go:117] "RemoveContainer" containerID="ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648" Jul 21 08:50:37.350541 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.350513 2553 scope.go:117] "RemoveContainer" containerID="7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4" Jul 21 08:50:37.359551 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.359527 2553 scope.go:117] "RemoveContainer" containerID="ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648" Jul 21 08:50:37.359827 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:50:37.359808 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648\": container with ID starting with ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648 not found: ID does not exist" containerID="ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648" Jul 21 08:50:37.359902 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.359841 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648"} err="failed to get container status \"ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648\": rpc error: code = NotFound desc = could not find container \"ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648\": container with ID starting with ae9cea58275534d3f76ee822a8a9c5ba9a202a5b899b7880e0254cdb8b8aa648 not found: ID does not exist" Jul 21 08:50:37.359902 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.359869 2553 scope.go:117] "RemoveContainer" containerID="7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4" Jul 21 08:50:37.360149 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:50:37.360131 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4\": container with ID starting with 7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4 not found: ID does not exist" containerID="7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4" Jul 21 08:50:37.360186 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.360157 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4"} err="failed to get container status \"7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4\": rpc error: code = NotFound desc = could not find container \"7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4\": container with ID starting with 7ce7500910e80713f82f1f3a13281406caeba0ab34fa93e560300172c4d1efd4 not found: ID does not exist" Jul 21 08:50:37.361451 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.361431 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5"] Jul 21 08:50:37.365419 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:37.365399 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-68712b6b-openshift-default-7d84dbfb95-x9dx5"] Jul 21 08:50:38.674835 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:38.674799 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" path="/var/lib/kubelet/pods/15f9a82a-77d8-46e0-8175-b6e8b65232ad/volumes" Jul 21 08:50:51.264010 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.263972 2553 patch_prober.go:28] interesting pod/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:47522->10.134.0.51:15021: read: connection reset by peer" start-of-body= Jul 21 08:50:51.264445 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.264086 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:47522->10.134.0.51:15021: read: connection reset by peer" Jul 21 08:50:51.417025 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.416989 2553 generic.go:356] "Generic (PLEG): container finished" podID="ee641070-3cb0-4047-a291-09bac166fe2f" containerID="a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39" exitCode=0 Jul 21 08:50:51.417200 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.417067 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" event={"ID":"ee641070-3cb0-4047-a291-09bac166fe2f","Type":"ContainerDied","Data":"a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39"} Jul 21 08:50:51.417509 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.417496 2553 scope.go:117] "RemoveContainer" containerID="a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39" Jul 21 08:50:51.418572 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.418541 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:51.418666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.418626 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:51.418715 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:51.418670 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:50:52.423842 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:52.423805 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" event={"ID":"ee641070-3cb0-4047-a291-09bac166fe2f","Type":"ContainerStarted","Data":"1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8"} Jul 21 08:50:52.853651 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:52.853570 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:52.853852 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:52.853830 2553 patch_prober.go:28] interesting pod/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 08:50:52.853903 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:52.853878 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 08:50:53.853929 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:53.853894 2553 patch_prober.go:28] interesting pod/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 08:50:53.854389 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:53.853951 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 08:50:54.853680 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:54.853645 2553 patch_prober.go:28] interesting pod/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 08:50:54.853879 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:54.853701 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 08:50:55.854186 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:55.854139 2553 patch_prober.go:28] interesting pod/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 08:50:55.854605 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:55.854207 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 08:50:56.857602 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:56.857570 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:56.858101 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:56.857961 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:50:56.858710 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:50:56.858691 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:51:15.095337 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.092792 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2"] Jul 21 08:51:15.095337 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.094281 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" containerID="cri-o://1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8" gracePeriod=30 Jul 21 08:51:15.918207 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.918170 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm"] Jul 21 08:51:15.918602 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.918588 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" Jul 21 08:51:15.918653 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.918604 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" Jul 21 08:51:15.918688 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.918680 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" Jul 21 08:51:15.918726 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.918690 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" Jul 21 08:51:15.921633 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.921617 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:15.923910 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.923887 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-81c79699-openshift-default-dockercfg-6h7t8\"" Jul 21 08:51:15.936397 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:15.936374 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm"] Jul 21 08:51:16.094349 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-token\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094528 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094377 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-envoy\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094528 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094424 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-socket\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094528 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094446 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-podinfo\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094528 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094470 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istiod-ca-cert\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094528 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094494 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-certs\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094692 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094545 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-data\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094692 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094565 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-credential-socket\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.094692 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.094585 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6flw\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-kube-api-access-c6flw\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.195617 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195520 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-token\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.195617 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195586 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-envoy\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195651 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-socket\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195675 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-podinfo\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istiod-ca-cert\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195751 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-certs\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195782 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-data\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195817 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-credential-socket\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.195846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6flw\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-kube-api-access-c6flw\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196171 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.196074 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-socket\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196516 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.196202 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-certs\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196516 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.196237 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-data\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196516 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.196260 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-credential-socket\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.196645 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.196627 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istiod-ca-cert\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.198064 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.198042 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-envoy\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.198323 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.198293 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-podinfo\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.203785 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.203759 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-token\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.203895 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.203793 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6flw\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-kube-api-access-c6flw\") pod \"e2e-ait-81c79699-openshift-default-8f469c687-55jvm\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.232566 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.232532 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:16.375013 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.374985 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm"] Jul 21 08:51:16.377203 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:51:16.377178 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8676bb7_b0e3_49c5_ba6b_1870ed03b8a5.slice/crio-06d1ad0d0dea10953d62e891bbe3ef83b73b90412f4157df590434fe3e026364 WatchSource:0}: Error finding container 06d1ad0d0dea10953d62e891bbe3ef83b73b90412f4157df590434fe3e026364: Status 404 returned error can't find the container with id 06d1ad0d0dea10953d62e891bbe3ef83b73b90412f4157df590434fe3e026364 Jul 21 08:51:16.379485 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.379446 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:51:16.379570 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.379546 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:51:16.379611 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.379591 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:51:16.548577 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.548481 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" event={"ID":"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5","Type":"ContainerStarted","Data":"5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395"} Jul 21 08:51:16.548577 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.548555 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" event={"ID":"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5","Type":"ContainerStarted","Data":"06d1ad0d0dea10953d62e891bbe3ef83b73b90412f4157df590434fe3e026364"} Jul 21 08:51:16.567767 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:16.567696 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" podStartSLOduration=1.5676766290000002 podStartE2EDuration="1.567676629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:51:16.566858502 +0000 UTC m=+1932.452529053" watchObservedRunningTime="2026-07-21 08:51:16.567676629 +0000 UTC m=+1932.453347177" Jul 21 08:51:17.233374 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:17.233346 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:17.238782 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:17.238755 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:17.554765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:17.554672 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:17.555741 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:17.555719 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:20.349693 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.349668 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:51:20.438339 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438261 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-socket\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438530 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438348 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee641070-3cb0-4047-a291-09bac166fe2f-istio-podinfo\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438530 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438427 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee641070-3cb0-4047-a291-09bac166fe2f-istiod-ca-cert\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438530 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438460 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-certs\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438530 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438496 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-credential-socket\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438530 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438522 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-data\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438583 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-envoy\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438598 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:51:20.438877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438613 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nb6z\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-kube-api-access-9nb6z\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438643 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-istio-token\") pod \"ee641070-3cb0-4047-a291-09bac166fe2f\" (UID: \"ee641070-3cb0-4047-a291-09bac166fe2f\") " Jul 21 08:51:20.438877 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438758 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:51:20.439162 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438903 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:51:20.439162 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.438862 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee641070-3cb0-4047-a291-09bac166fe2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:51:20.439162 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.439034 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee641070-3cb0-4047-a291-09bac166fe2f-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.439162 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.439054 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.439162 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.439068 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.439620 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.439163 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.439620 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.439550 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:51:20.440737 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.440710 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee641070-3cb0-4047-a291-09bac166fe2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 08:51:20.440988 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.440972 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:51:20.441260 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.441233 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-kube-api-access-9nb6z" (OuterVolumeSpecName: "kube-api-access-9nb6z") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "kube-api-access-9nb6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:51:20.441260 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.441236 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee641070-3cb0-4047-a291-09bac166fe2f" (UID: "ee641070-3cb0-4047-a291-09bac166fe2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:51:20.540347 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.540267 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.540347 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.540342 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee641070-3cb0-4047-a291-09bac166fe2f-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.540347 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.540355 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nb6z\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-kube-api-access-9nb6z\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.540592 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.540368 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee641070-3cb0-4047-a291-09bac166fe2f-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.540592 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.540380 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee641070-3cb0-4047-a291-09bac166fe2f-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:51:20.570378 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.570343 2553 generic.go:356] "Generic (PLEG): container finished" podID="ee641070-3cb0-4047-a291-09bac166fe2f" containerID="1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8" exitCode=0 Jul 21 08:51:20.570564 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.570393 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" Jul 21 08:51:20.570564 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.570418 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" event={"ID":"ee641070-3cb0-4047-a291-09bac166fe2f","Type":"ContainerDied","Data":"1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8"} Jul 21 08:51:20.570564 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.570496 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2" event={"ID":"ee641070-3cb0-4047-a291-09bac166fe2f","Type":"ContainerDied","Data":"6984af358a1a45f6f658d43d71622edc115ede2f114d4439aed1936f11b0793f"} Jul 21 08:51:20.570564 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.570526 2553 scope.go:117] "RemoveContainer" containerID="1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8" Jul 21 08:51:20.580253 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.580237 2553 scope.go:117] "RemoveContainer" containerID="a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39" Jul 21 08:51:20.588971 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.588951 2553 scope.go:117] "RemoveContainer" containerID="1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8" Jul 21 08:51:20.589208 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:51:20.589185 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8\": container with ID starting with 1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8 not found: ID does not exist" containerID="1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8" Jul 21 08:51:20.589274 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.589215 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8"} err="failed to get container status \"1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8\": rpc error: code = NotFound desc = could not find container \"1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8\": container with ID starting with 1980f7e5a9c39e0894f85638429db7106209ca37b0c53967674b6c318adc8ad8 not found: ID does not exist" Jul 21 08:51:20.589274 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.589232 2553 scope.go:117] "RemoveContainer" containerID="a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39" Jul 21 08:51:20.589491 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:51:20.589474 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39\": container with ID starting with a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39 not found: ID does not exist" containerID="a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39" Jul 21 08:51:20.589534 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.589496 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39"} err="failed to get container status \"a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39\": rpc error: code = NotFound desc = could not find container \"a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39\": container with ID starting with a4a7793cedd1046718b68c0473bee7480c27d7f669001901bbe342b42d658e39 not found: ID does not exist" Jul 21 08:51:20.593480 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.593459 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2"] Jul 21 08:51:20.600490 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.600469 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-500e3541-gw-openshift-default-b8b7d586c-p2zw2"] Jul 21 08:51:20.674817 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:20.674767 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" path="/var/lib/kubelet/pods/ee641070-3cb0-4047-a291-09bac166fe2f/volumes" Jul 21 08:51:32.556910 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.556819 2553 patch_prober.go:28] interesting pod/e2e-ait-81c79699-openshift-default-8f469c687-55jvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 08:51:32.557355 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.556904 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 08:51:32.642984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.642947 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerID="5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395" exitCode=0 Jul 21 08:51:32.643155 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.642992 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" event={"ID":"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5","Type":"ContainerDied","Data":"5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395"} Jul 21 08:51:32.643435 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.643420 2553 scope.go:117] "RemoveContainer" containerID="5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395" Jul 21 08:51:32.644525 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.644490 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:51:32.644644 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.644577 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:51:32.644644 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:32.644626 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:51:33.650240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:33.650205 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" event={"ID":"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5","Type":"ContainerStarted","Data":"32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf"} Jul 21 08:51:34.233542 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:34.233507 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:34.233871 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:34.233848 2553 patch_prober.go:28] interesting pod/e2e-ait-81c79699-openshift-default-8f469c687-55jvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 08:51:34.233928 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:34.233898 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 08:51:35.233630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:35.233590 2553 patch_prober.go:28] interesting pod/e2e-ait-81c79699-openshift-default-8f469c687-55jvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 08:51:35.234036 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:35.233646 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 08:51:36.233361 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:36.233284 2553 patch_prober.go:28] interesting pod/e2e-ait-81c79699-openshift-default-8f469c687-55jvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 08:51:36.233623 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:36.233387 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 08:51:37.233444 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:37.233411 2553 patch_prober.go:28] interesting pod/e2e-ait-81c79699-openshift-default-8f469c687-55jvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 21 08:51:37.233827 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:37.233468 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 21 08:51:38.237529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:38.237499 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:38.237955 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:38.237940 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:51:38.238816 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:51:38.238795 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:52:00.093470 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.093425 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm"] Jul 21 08:52:00.093985 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.093695 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" containerID="cri-o://32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf" gracePeriod=30 Jul 21 08:52:00.867155 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867124 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4"] Jul 21 08:52:00.867590 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867560 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" Jul 21 08:52:00.867590 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867581 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" Jul 21 08:52:00.867765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867598 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" Jul 21 08:52:00.867765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867607 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" Jul 21 08:52:00.867765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867624 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" Jul 21 08:52:00.867765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867633 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="15f9a82a-77d8-46e0-8175-b6e8b65232ad" containerName="istio-proxy" Jul 21 08:52:00.867765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.867713 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" Jul 21 08:52:00.869976 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.869960 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:00.872450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.872430 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-84fe4740-openshift-default-dockercfg-498sj\"" Jul 21 08:52:00.884542 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:00.884514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4"] Jul 21 08:52:01.039614 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039559 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7396ab1-ef93-4cdc-a513-738b6b564330-istiod-ca-cert\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039632 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-certs\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-credential-socket\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039678 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-envoy\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-data\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039796 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039728 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-token\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039965 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039799 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rq2\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-kube-api-access-m8rq2\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039965 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039825 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-podinfo\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.039965 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.039849 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-socket\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.140996 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-certs\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141045 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-credential-socket\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141070 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-envoy\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141095 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141097 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-data\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141124 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-token\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141183 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rq2\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-kube-api-access-m8rq2\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141291 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-podinfo\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141364 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-socket\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141400 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7396ab1-ef93-4cdc-a513-738b6b564330-istiod-ca-cert\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141483 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-certs\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.141771 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141637 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-data\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.142088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141806 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-socket\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.142088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.141911 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-credential-socket\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.142340 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.142301 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7396ab1-ef93-4cdc-a513-738b6b564330-istiod-ca-cert\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.143536 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.143517 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-envoy\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.143958 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.143941 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-podinfo\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.150117 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.150087 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rq2\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-kube-api-access-m8rq2\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.150599 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.150580 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-token\") pod \"e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.180787 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.180752 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:01.325058 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.325031 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4"] Jul 21 08:52:01.327140 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:52:01.327109 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7396ab1_ef93_4cdc_a513_738b6b564330.slice/crio-2ff244f3d9bb6792d05f52e45cdce41b29e10fac8bc63cd8b336e7a5fa0e99de WatchSource:0}: Error finding container 2ff244f3d9bb6792d05f52e45cdce41b29e10fac8bc63cd8b336e7a5fa0e99de: Status 404 returned error can't find the container with id 2ff244f3d9bb6792d05f52e45cdce41b29e10fac8bc63cd8b336e7a5fa0e99de Jul 21 08:52:01.329325 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.329271 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:52:01.329446 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.329372 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:52:01.329446 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.329422 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:52:01.794822 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.794792 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" event={"ID":"a7396ab1-ef93-4cdc-a513-738b6b564330","Type":"ContainerStarted","Data":"cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136"} Jul 21 08:52:01.795003 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.794842 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" event={"ID":"a7396ab1-ef93-4cdc-a513-738b6b564330","Type":"ContainerStarted","Data":"2ff244f3d9bb6792d05f52e45cdce41b29e10fac8bc63cd8b336e7a5fa0e99de"} Jul 21 08:52:01.816098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:01.816052 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" podStartSLOduration=1.816037707 podStartE2EDuration="1.816037707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:52:01.814918312 +0000 UTC m=+1977.700588890" watchObservedRunningTime="2026-07-21 08:52:01.816037707 +0000 UTC m=+1977.701708255" Jul 21 08:52:02.180996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:02.180953 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:02.186359 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:02.186331 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:02.799950 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:02.799924 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:02.800989 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:02.800965 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:05.360304 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.360232 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:52:05.381217 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381190 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-data\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381298 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-podinfo\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381356 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istiod-ca-cert\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381381 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-certs\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381409 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-socket\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381500 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-envoy\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381532 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-token\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381613 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6flw\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-kube-api-access-c6flw\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.381670 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381641 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-credential-socket\") pod \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\" (UID: \"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5\") " Jul 21 08:52:05.384456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.381818 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:05.384456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.382017 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:05.384456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.382157 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.384456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.382175 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.384456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.382568 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:05.384456 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.382821 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:52:05.384779 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.384605 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:05.385655 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.385510 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 08:52:05.385655 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.385526 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:52:05.385655 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.385614 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:05.385655 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.385611 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-kube-api-access-c6flw" (OuterVolumeSpecName: "kube-api-access-c6flw") pod "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" (UID: "d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5"). InnerVolumeSpecName "kube-api-access-c6flw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:52:05.483027 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.482987 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.483027 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.483020 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.483027 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.483032 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.483284 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.483044 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.483284 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.483056 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.483284 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.483068 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6flw\" (UniqueName: \"kubernetes.io/projected/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-kube-api-access-c6flw\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.483284 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.483079 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:05.817039 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.817007 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerID="32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf" exitCode=0 Jul 21 08:52:05.817200 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.817060 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" Jul 21 08:52:05.817200 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.817080 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" event={"ID":"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5","Type":"ContainerDied","Data":"32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf"} Jul 21 08:52:05.817200 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.817123 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm" event={"ID":"d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5","Type":"ContainerDied","Data":"06d1ad0d0dea10953d62e891bbe3ef83b73b90412f4157df590434fe3e026364"} Jul 21 08:52:05.817200 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.817146 2553 scope.go:117] "RemoveContainer" containerID="32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf" Jul 21 08:52:05.826737 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.826720 2553 scope.go:117] "RemoveContainer" containerID="5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395" Jul 21 08:52:05.834943 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.834918 2553 scope.go:117] "RemoveContainer" containerID="32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf" Jul 21 08:52:05.835207 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:52:05.835190 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf\": container with ID starting with 32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf not found: ID does not exist" containerID="32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf" Jul 21 08:52:05.835270 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.835216 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf"} err="failed to get container status \"32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf\": rpc error: code = NotFound desc = could not find container \"32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf\": container with ID starting with 32a154eb84929f9c1da53bbea55a156191379ace1cd55a36f77c5509c457bbaf not found: ID does not exist" Jul 21 08:52:05.835343 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.835273 2553 scope.go:117] "RemoveContainer" containerID="5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395" Jul 21 08:52:05.835553 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:52:05.835532 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395\": container with ID starting with 5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395 not found: ID does not exist" containerID="5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395" Jul 21 08:52:05.835612 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.835560 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395"} err="failed to get container status \"5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395\": rpc error: code = NotFound desc = could not find container \"5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395\": container with ID starting with 5c634754b1e6110551a6790f9e14d4dcd2d36074ce3a4f9dd337a4634331c395 not found: ID does not exist" Jul 21 08:52:05.844618 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.844593 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm"] Jul 21 08:52:05.851792 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:05.851766 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-81c79699-openshift-default-8f469c687-55jvm"] Jul 21 08:52:06.674419 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:06.674386 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" path="/var/lib/kubelet/pods/d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5/volumes" Jul 21 08:52:16.875546 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:16.875509 2553 generic.go:356] "Generic (PLEG): container finished" podID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerID="cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136" exitCode=0 Jul 21 08:52:16.875944 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:16.875588 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" event={"ID":"a7396ab1-ef93-4cdc-a513-738b6b564330","Type":"ContainerDied","Data":"cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136"} Jul 21 08:52:16.875990 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:16.875982 2553 scope.go:117] "RemoveContainer" containerID="cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136" Jul 21 08:52:16.877018 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:16.876990 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:52:16.877079 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:16.877058 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:52:16.877115 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:16.877087 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:52:17.799999 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:17.799946 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:17.882767 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:17.882712 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" event={"ID":"a7396ab1-ef93-4cdc-a513-738b6b564330","Type":"ContainerStarted","Data":"113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d"} Jul 21 08:52:18.181207 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:18.181164 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:18.182196 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:18.182168 2553 patch_prober.go:28] interesting pod/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 08:52:18.182344 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:18.182234 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 08:52:19.181404 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:19.181363 2553 patch_prober.go:28] interesting pod/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 08:52:19.181795 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:19.181433 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 08:52:20.181666 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:20.181626 2553 patch_prober.go:28] interesting pod/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 08:52:20.182055 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:20.181700 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 08:52:21.182232 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:21.182199 2553 patch_prober.go:28] interesting pod/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 08:52:21.182768 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:21.182263 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 08:52:22.185393 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:22.185358 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:22.185769 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:22.185700 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:22.186346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:22.186302 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:43.380896 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:43.380857 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4"] Jul 21 08:52:43.381481 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:43.381073 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" containerID="cri-o://113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d" gracePeriod=30 Jul 21 08:52:48.650253 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.650185 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:48.706141 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706111 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-token\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706329 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706152 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-podinfo\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706329 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706275 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-certs\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706423 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706340 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-socket\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706423 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706368 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-credential-socket\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706423 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706398 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-data\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706578 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706440 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7396ab1-ef93-4cdc-a513-738b6b564330-istiod-ca-cert\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706578 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706468 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8rq2\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-kube-api-access-m8rq2\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706578 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706494 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-envoy\") pod \"a7396ab1-ef93-4cdc-a513-738b6b564330\" (UID: \"a7396ab1-ef93-4cdc-a513-738b6b564330\") " Jul 21 08:52:48.706725 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706589 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:48.706725 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.706602 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:48.707082 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.707042 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:48.707275 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.707251 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.707364 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.707284 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.707364 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.707341 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.707481 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.707437 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:48.709155 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.707551 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7396ab1-ef93-4cdc-a513-738b6b564330-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:52:48.709601 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.709567 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:52:48.709743 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.709706 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:52:48.711394 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.709822 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-kube-api-access-m8rq2" (OuterVolumeSpecName: "kube-api-access-m8rq2") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "kube-api-access-m8rq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:52:48.715864 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.715835 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7396ab1-ef93-4cdc-a513-738b6b564330" (UID: "a7396ab1-ef93-4cdc-a513-738b6b564330"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 08:52:48.808127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.808092 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.808127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.808122 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.808127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.808131 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.808383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.808140 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7396ab1-ef93-4cdc-a513-738b6b564330-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.808383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.808149 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rq2\" (UniqueName: \"kubernetes.io/projected/a7396ab1-ef93-4cdc-a513-738b6b564330-kube-api-access-m8rq2\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:48.808383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:48.808159 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7396ab1-ef93-4cdc-a513-738b6b564330-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:52:49.046812 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.046776 2553 generic.go:356] "Generic (PLEG): container finished" podID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerID="113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d" exitCode=0 Jul 21 08:52:49.046984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.046833 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" Jul 21 08:52:49.046984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.046854 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" event={"ID":"a7396ab1-ef93-4cdc-a513-738b6b564330","Type":"ContainerDied","Data":"113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d"} Jul 21 08:52:49.046984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.046934 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4" event={"ID":"a7396ab1-ef93-4cdc-a513-738b6b564330","Type":"ContainerDied","Data":"2ff244f3d9bb6792d05f52e45cdce41b29e10fac8bc63cd8b336e7a5fa0e99de"} Jul 21 08:52:49.046984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.046959 2553 scope.go:117] "RemoveContainer" containerID="113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d" Jul 21 08:52:49.056665 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.056647 2553 scope.go:117] "RemoveContainer" containerID="cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136" Jul 21 08:52:49.065180 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.065165 2553 scope.go:117] "RemoveContainer" containerID="113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d" Jul 21 08:52:49.065459 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:52:49.065442 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d\": container with ID starting with 113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d not found: ID does not exist" containerID="113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d" Jul 21 08:52:49.065509 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.065469 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d"} err="failed to get container status \"113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d\": rpc error: code = NotFound desc = could not find container \"113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d\": container with ID starting with 113fa8f3dc51cae159492f82256d85839b190a1e95889b757b3220e2a3c6a67d not found: ID does not exist" Jul 21 08:52:49.065509 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.065487 2553 scope.go:117] "RemoveContainer" containerID="cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136" Jul 21 08:52:49.065698 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:52:49.065683 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136\": container with ID starting with cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136 not found: ID does not exist" containerID="cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136" Jul 21 08:52:49.065737 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.065703 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136"} err="failed to get container status \"cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136\": rpc error: code = NotFound desc = could not find container \"cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136\": container with ID starting with cfea69a003aad9cc0e53bed6bd5cce00ccb58fea701ba34f404844eb311bc136 not found: ID does not exist" Jul 21 08:52:49.072275 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.072248 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4"] Jul 21 08:52:49.076505 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:49.076482 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-84fe4740-openshift-default-795c85bd9f-2vsq4"] Jul 21 08:52:50.674093 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:52:50.674058 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" path="/var/lib/kubelet/pods/a7396ab1-ef93-4cdc-a513-738b6b564330/volumes" Jul 21 08:54:05.114683 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:05.114647 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:54:05.117996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:05.117973 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:54:25.050443 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.050405 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9"] Jul 21 08:54:25.051000 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.050987 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" Jul 21 08:54:25.051074 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051006 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" Jul 21 08:54:25.051074 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051025 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" Jul 21 08:54:25.051074 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051033 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" Jul 21 08:54:25.051074 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051046 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" Jul 21 08:54:25.051074 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051055 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" Jul 21 08:54:25.051358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051082 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" Jul 21 08:54:25.051358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051091 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" Jul 21 08:54:25.051358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051183 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" Jul 21 08:54:25.051358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051198 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8676bb7-b0e3-49c5-ba6b-1870ed03b8a5" containerName="istio-proxy" Jul 21 08:54:25.051358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051209 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" Jul 21 08:54:25.051358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051219 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7396ab1-ef93-4cdc-a513-738b6b564330" containerName="istio-proxy" Jul 21 08:54:25.051358 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.051230 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee641070-3cb0-4047-a291-09bac166fe2f" containerName="istio-proxy" Jul 21 08:54:25.054484 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.054463 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.057922 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.057902 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c2dd45b4-openshift-default-dockercfg-4vcw2\"" Jul 21 08:54:25.123705 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123671 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-envoy\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.123886 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123727 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-podinfo\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.123886 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-token\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.123886 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123853 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istiod-ca-cert\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.123996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123891 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-socket\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.123996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123943 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-data\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.123996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123970 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-credential-socket\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.123996 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.123985 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-certs\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.124127 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.124002 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkvj\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-kube-api-access-jvkvj\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.141348 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.141270 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9"] Jul 21 08:54:25.225442 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225402 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-socket\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.225640 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225455 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-data\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.225711 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225633 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-credential-socket\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.225711 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-certs\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.225711 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225693 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkvj\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-kube-api-access-jvkvj\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.225883 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225735 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-envoy\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.225883 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225792 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-podinfo\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.225883 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225800 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-data\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.226048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225884 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-socket\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.226048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225971 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-certs\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.226048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.225979 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-token\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.226048 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.226021 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istiod-ca-cert\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.226249 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.226054 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-credential-socket\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.226734 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.226712 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istiod-ca-cert\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.228206 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.228186 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-envoy\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.228444 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.228427 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-podinfo\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.256569 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.256537 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkvj\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-kube-api-access-jvkvj\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.257424 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.257404 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-token\") pod \"e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.364534 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.364423 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:25.508993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.508958 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9"] Jul 21 08:54:25.512700 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:54:25.512660 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cccbfff_cfe1_4575_ab53_8cf98d14d6e7.slice/crio-32cf291a4b7dfc8f3398c44afd5333d2e47861ed76388f8cb26c4803b0bcb18c WatchSource:0}: Error finding container 32cf291a4b7dfc8f3398c44afd5333d2e47861ed76388f8cb26c4803b0bcb18c: Status 404 returned error can't find the container with id 32cf291a4b7dfc8f3398c44afd5333d2e47861ed76388f8cb26c4803b0bcb18c Jul 21 08:54:25.514967 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.514933 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:54:25.515053 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.515002 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:54:25.515053 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.515030 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:54:25.549187 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:25.549153 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" event={"ID":"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7","Type":"ContainerStarted","Data":"32cf291a4b7dfc8f3398c44afd5333d2e47861ed76388f8cb26c4803b0bcb18c"} Jul 21 08:54:26.556264 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:26.556224 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" event={"ID":"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7","Type":"ContainerStarted","Data":"5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de"} Jul 21 08:54:26.580387 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:26.580337 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" podStartSLOduration=1.58028933 podStartE2EDuration="1.58028933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:54:26.578809589 +0000 UTC m=+2122.464480136" watchObservedRunningTime="2026-07-21 08:54:26.58028933 +0000 UTC m=+2122.465959877" Jul 21 08:54:27.365398 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:27.365358 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:27.371289 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:27.371257 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:27.563080 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:27.563049 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:27.563841 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:27.563822 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:41.645485 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:41.645446 2553 generic.go:356] "Generic (PLEG): container finished" podID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerID="5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de" exitCode=0 Jul 21 08:54:41.646032 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:41.645490 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" event={"ID":"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7","Type":"ContainerDied","Data":"5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de"} Jul 21 08:54:41.646032 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:41.645998 2553 scope.go:117] "RemoveContainer" containerID="5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de" Jul 21 08:54:41.647160 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:41.647121 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:54:41.647266 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:41.647200 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:54:41.647266 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:41.647239 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:54:42.563294 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:42.563258 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:42.652965 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:42.652928 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" event={"ID":"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7","Type":"ContainerStarted","Data":"32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2"} Jul 21 08:54:43.364979 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:43.364941 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:43.366400 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:43.366376 2553 patch_prober.go:28] interesting pod/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 08:54:43.366467 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:43.366443 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 08:54:44.365827 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:44.365785 2553 patch_prober.go:28] interesting pod/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 08:54:44.366260 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:44.365860 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 08:54:45.365681 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:45.365644 2553 patch_prober.go:28] interesting pod/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 08:54:45.365875 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:45.365703 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 08:54:46.365661 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:46.365620 2553 patch_prober.go:28] interesting pod/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 21 08:54:46.365844 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:46.365683 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 21 08:54:47.368730 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:47.368699 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:47.369204 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:47.368982 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:54:47.369824 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:54:47.369808 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:55:02.154712 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:02.154612 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9"] Jul 21 08:55:02.155247 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:02.154876 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" containerID="cri-o://32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2" gracePeriod=30 Jul 21 08:55:02.371366 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:02.371326 2553 patch_prober.go:28] interesting pod/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 08:55:02.371553 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:02.371410 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 08:55:07.416599 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.416571 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:55:07.548022 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.547934 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-socket\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548022 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.547983 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-token\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548022 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548007 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-certs\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548031 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-envoy\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548086 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istiod-ca-cert\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548112 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-podinfo\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548136 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-credential-socket\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548195 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-data\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548271 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548260 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvkvj\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-kube-api-access-jvkvj\") pod \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\" (UID: \"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7\") " Jul 21 08:55:07.548630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548301 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:55:07.548630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548354 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:55:07.548630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548589 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:55:07.548630 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548618 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.548843 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548642 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.548843 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.548748 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:55:07.549163 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.549139 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:55:07.550505 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.550479 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-kube-api-access-jvkvj" (OuterVolumeSpecName: "kube-api-access-jvkvj") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "kube-api-access-jvkvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:55:07.550623 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.550603 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:55:07.550857 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.550835 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:55:07.550928 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.550901 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" (UID: "0cccbfff-cfe1-4575-ab53-8cf98d14d6e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 08:55:07.649508 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.649470 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.649508 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.649502 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.649508 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.649511 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.649508 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.649519 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.649773 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.649527 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.649773 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.649536 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.649773 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.649545 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkvj\" (UniqueName: \"kubernetes.io/projected/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7-kube-api-access-jvkvj\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:55:07.784922 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.784884 2553 generic.go:356] "Generic (PLEG): container finished" podID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerID="32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2" exitCode=0 Jul 21 08:55:07.785096 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.784938 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" Jul 21 08:55:07.785096 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.784977 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" event={"ID":"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7","Type":"ContainerDied","Data":"32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2"} Jul 21 08:55:07.785096 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.785034 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9" event={"ID":"0cccbfff-cfe1-4575-ab53-8cf98d14d6e7","Type":"ContainerDied","Data":"32cf291a4b7dfc8f3398c44afd5333d2e47861ed76388f8cb26c4803b0bcb18c"} Jul 21 08:55:07.785096 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.785060 2553 scope.go:117] "RemoveContainer" containerID="32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2" Jul 21 08:55:07.794795 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.794775 2553 scope.go:117] "RemoveContainer" containerID="5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de" Jul 21 08:55:07.803134 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.803113 2553 scope.go:117] "RemoveContainer" containerID="32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2" Jul 21 08:55:07.803440 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:55:07.803419 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2\": container with ID starting with 32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2 not found: ID does not exist" containerID="32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2" Jul 21 08:55:07.803518 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.803450 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2"} err="failed to get container status \"32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2\": rpc error: code = NotFound desc = could not find container \"32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2\": container with ID starting with 32241a9000c0e391e8dff09a97be011af1b676479dd0539cdab4da14edbb68f2 not found: ID does not exist" Jul 21 08:55:07.803518 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.803467 2553 scope.go:117] "RemoveContainer" containerID="5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de" Jul 21 08:55:07.803710 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:55:07.803693 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de\": container with ID starting with 5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de not found: ID does not exist" containerID="5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de" Jul 21 08:55:07.803761 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.803718 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de"} err="failed to get container status \"5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de\": rpc error: code = NotFound desc = could not find container \"5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de\": container with ID starting with 5de6e5a4034b1e9968197d8982ce825fe18eb60dfae8e8d495e480156d17b2de not found: ID does not exist" Jul 21 08:55:07.807261 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.807241 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9"] Jul 21 08:55:07.813579 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:07.813557 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c2dd45b4-openshift-default-5d774bbd95-f6vt9"] Jul 21 08:55:08.673688 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:08.673657 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" path="/var/lib/kubelet/pods/0cccbfff-cfe1-4575-ab53-8cf98d14d6e7/volumes" Jul 21 08:55:22.202965 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.202920 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x"] Jul 21 08:55:22.203529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.203508 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" Jul 21 08:55:22.203622 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.203532 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" Jul 21 08:55:22.203622 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.203562 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" Jul 21 08:55:22.203622 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.203572 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" Jul 21 08:55:22.203770 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.203686 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" Jul 21 08:55:22.203770 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.203700 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cccbfff-cfe1-4575-ab53-8cf98d14d6e7" containerName="istio-proxy" Jul 21 08:55:22.206872 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.206849 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.209331 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.209290 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f491e220-openshift-default-dockercfg-b6j9v\"" Jul 21 08:55:22.223156 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.223128 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x"] Jul 21 08:55:22.392152 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392117 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istiod-ca-cert\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392163 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-certs\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392206 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-token\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392245 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-socket\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392354 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392320 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-data\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392501 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392354 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x228\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-kube-api-access-9x228\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392501 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392384 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-podinfo\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392501 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392446 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-envoy\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.392501 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.392492 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-credential-socket\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.493810 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.493719 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-socket\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.493810 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.493760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-data\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.493810 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.493790 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x228\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-kube-api-access-9x228\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.493830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-podinfo\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.493877 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-envoy\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.493932 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-credential-socket\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494098 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.493969 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istiod-ca-cert\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494293 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.494171 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-socket\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494383 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.494344 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-credential-socket\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494429 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.494418 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-certs\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494507 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.494488 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-token\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494721 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.494702 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-data\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494780 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.494717 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istiod-ca-cert\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.494913 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.494889 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-certs\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.496506 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.496472 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-envoy\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.496692 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.496676 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-podinfo\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.505599 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.505573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-token\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.506081 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.506062 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x228\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-kube-api-access-9x228\") pod \"e2e-mt-f491e220-openshift-default-595c848747-45g7x\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.517017 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.516995 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:22.660948 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.660920 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x"] Jul 21 08:55:22.663793 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:55:22.663766 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebfd439_5fda_48f0_b11b_b9b10ab6918d.slice/crio-c978c32a1848de20fdde4ccc795a57ecab83de2b4d1a2839bd41b6f9f53983b3 WatchSource:0}: Error finding container c978c32a1848de20fdde4ccc795a57ecab83de2b4d1a2839bd41b6f9f53983b3: Status 404 returned error can't find the container with id c978c32a1848de20fdde4ccc795a57ecab83de2b4d1a2839bd41b6f9f53983b3 Jul 21 08:55:22.666011 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.665978 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:55:22.666085 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.666051 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:55:22.666122 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.666082 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:55:22.866663 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.866567 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" event={"ID":"cebfd439-5fda-48f0-b11b-b9b10ab6918d","Type":"ContainerStarted","Data":"48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce"} Jul 21 08:55:22.866663 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.866637 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" event={"ID":"cebfd439-5fda-48f0-b11b-b9b10ab6918d","Type":"ContainerStarted","Data":"c978c32a1848de20fdde4ccc795a57ecab83de2b4d1a2839bd41b6f9f53983b3"} Jul 21 08:55:22.885743 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:22.885680 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" podStartSLOduration=0.885660024 podStartE2EDuration="885.660024ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:55:22.884656735 +0000 UTC m=+2178.770327282" watchObservedRunningTime="2026-07-21 08:55:22.885660024 +0000 UTC m=+2178.771330574" Jul 21 08:55:23.517908 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:23.517868 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:23.522731 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:23.522684 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:23.871615 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:23.871516 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:23.872652 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:23.872632 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:25.073432 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.073394 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b"] Jul 21 08:55:25.076583 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.076561 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.080254 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.080233 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f491e220-serving-cert\"" Jul 21 08:55:25.091464 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.091438 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b"] Jul 21 08:55:25.121821 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.121783 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.122000 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.121844 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcec3e-1356-443c-90d6-bfd32a029aef-openshift-service-ca\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.122000 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.121943 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnkm\" (UniqueName: \"kubernetes.io/projected/d6fcec3e-1356-443c-90d6-bfd32a029aef-kube-api-access-kcnkm\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.223021 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.222989 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.223213 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.223054 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcec3e-1356-443c-90d6-bfd32a029aef-openshift-service-ca\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.223213 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.223100 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnkm\" (UniqueName: \"kubernetes.io/projected/d6fcec3e-1356-443c-90d6-bfd32a029aef-kube-api-access-kcnkm\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.223213 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:55:25.223176 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-serving-cert: secret "maas-api-e2e-mt-f491e220-serving-cert" not found Jul 21 08:55:25.223464 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:55:25.223258 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls podName:d6fcec3e-1356-443c-90d6-bfd32a029aef nodeName:}" failed. No retries permitted until 2026-07-21 08:55:25.723236402 +0000 UTC m=+2181.608906930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls") pod "maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" (UID: "d6fcec3e-1356-443c-90d6-bfd32a029aef") : secret "maas-api-e2e-mt-f491e220-serving-cert" not found Jul 21 08:55:25.223767 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.223749 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcec3e-1356-443c-90d6-bfd32a029aef-openshift-service-ca\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.236574 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.236539 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnkm\" (UniqueName: \"kubernetes.io/projected/d6fcec3e-1356-443c-90d6-bfd32a029aef-kube-api-access-kcnkm\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.729479 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.729445 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.732006 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.731982 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls\") pod \"maas-api-e2e-mt-f491e220-745b6f98f5-gln4b\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:25.986961 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:25.986883 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:26.138766 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:55:26.138734 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fcec3e_1356_443c_90d6_bfd32a029aef.slice/crio-5b4148c7232e1a45435f7dc6d6d00f30cc6e2849137603d27d25f83e2ffbb2ac WatchSource:0}: Error finding container 5b4148c7232e1a45435f7dc6d6d00f30cc6e2849137603d27d25f83e2ffbb2ac: Status 404 returned error can't find the container with id 5b4148c7232e1a45435f7dc6d6d00f30cc6e2849137603d27d25f83e2ffbb2ac Jul 21 08:55:26.139250 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:26.139224 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b"] Jul 21 08:55:26.140208 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:26.140180 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 08:55:26.887725 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:26.887636 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" event={"ID":"d6fcec3e-1356-443c-90d6-bfd32a029aef","Type":"ContainerStarted","Data":"d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58"} Jul 21 08:55:26.887725 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:26.887707 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" event={"ID":"d6fcec3e-1356-443c-90d6-bfd32a029aef","Type":"ContainerStarted","Data":"5b4148c7232e1a45435f7dc6d6d00f30cc6e2849137603d27d25f83e2ffbb2ac"} Jul 21 08:55:26.887958 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:26.887729 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:26.906722 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:26.906669 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" podStartSLOduration=1.5962467679999999 podStartE2EDuration="1.906652513s" totalImagesPullingTime="310.405745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:55:26.904887791 +0000 UTC m=+2182.790558339" watchObservedRunningTime="2026-07-21 08:55:26.906652513 +0000 UTC m=+2182.792323061" Jul 21 08:55:32.911132 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:32.911098 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:55:39.277402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.276414 2553 patch_prober.go:28] interesting pod/e2e-mt-f491e220-openshift-default-595c848747-45g7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:52372->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 21 08:55:39.277402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.276484 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:52372->10.134.0.55:15021: read: connection reset by peer" Jul 21 08:55:39.955624 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.955590 2553 generic.go:356] "Generic (PLEG): container finished" podID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerID="48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce" exitCode=0 Jul 21 08:55:39.955805 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.955661 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" event={"ID":"cebfd439-5fda-48f0-b11b-b9b10ab6918d","Type":"ContainerDied","Data":"48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce"} Jul 21 08:55:39.956103 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.956090 2553 scope.go:117] "RemoveContainer" containerID="48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce" Jul 21 08:55:39.957081 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.957051 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:55:39.957153 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.957139 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:55:39.957198 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:39.957185 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:55:40.962348 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:40.962295 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" event={"ID":"cebfd439-5fda-48f0-b11b-b9b10ab6918d","Type":"ContainerStarted","Data":"f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29"} Jul 21 08:55:41.517407 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:41.517363 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:41.517638 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:41.517613 2553 patch_prober.go:28] interesting pod/e2e-mt-f491e220-openshift-default-595c848747-45g7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 08:55:41.517687 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:41.517667 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 08:55:42.517595 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:42.517550 2553 patch_prober.go:28] interesting pod/e2e-mt-f491e220-openshift-default-595c848747-45g7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 08:55:42.517981 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:42.517628 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 08:55:43.518038 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:43.517999 2553 patch_prober.go:28] interesting pod/e2e-mt-f491e220-openshift-default-595c848747-45g7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 08:55:43.518577 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:43.518083 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 08:55:44.517801 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:44.517761 2553 patch_prober.go:28] interesting pod/e2e-mt-f491e220-openshift-default-595c848747-45g7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 08:55:44.517986 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:44.517823 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 08:55:45.520750 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:45.520721 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:45.521189 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:45.521046 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:55:45.521824 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:55:45.521811 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:56:39.225984 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.225948 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b"] Jul 21 08:56:39.226446 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.226179 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" podUID="d6fcec3e-1356-443c-90d6-bfd32a029aef" containerName="maas-api" containerID="cri-o://d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58" gracePeriod=30 Jul 21 08:56:39.483879 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.483806 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:56:39.649233 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.649202 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls\") pod \"d6fcec3e-1356-443c-90d6-bfd32a029aef\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " Jul 21 08:56:39.649450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.649272 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcec3e-1356-443c-90d6-bfd32a029aef-openshift-service-ca\") pod \"d6fcec3e-1356-443c-90d6-bfd32a029aef\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " Jul 21 08:56:39.649450 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.649346 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcnkm\" (UniqueName: \"kubernetes.io/projected/d6fcec3e-1356-443c-90d6-bfd32a029aef-kube-api-access-kcnkm\") pod \"d6fcec3e-1356-443c-90d6-bfd32a029aef\" (UID: \"d6fcec3e-1356-443c-90d6-bfd32a029aef\") " Jul 21 08:56:39.649693 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.649666 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fcec3e-1356-443c-90d6-bfd32a029aef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6fcec3e-1356-443c-90d6-bfd32a029aef" (UID: "d6fcec3e-1356-443c-90d6-bfd32a029aef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:56:39.651479 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.651457 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6fcec3e-1356-443c-90d6-bfd32a029aef" (UID: "d6fcec3e-1356-443c-90d6-bfd32a029aef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 08:56:39.651585 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.651504 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fcec3e-1356-443c-90d6-bfd32a029aef-kube-api-access-kcnkm" (OuterVolumeSpecName: "kube-api-access-kcnkm") pod "d6fcec3e-1356-443c-90d6-bfd32a029aef" (UID: "d6fcec3e-1356-443c-90d6-bfd32a029aef"). InnerVolumeSpecName "kube-api-access-kcnkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:56:39.751326 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.751215 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6fcec3e-1356-443c-90d6-bfd32a029aef-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:56:39.751326 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.751245 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6fcec3e-1356-443c-90d6-bfd32a029aef-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:56:39.751326 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:39.751258 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcnkm\" (UniqueName: \"kubernetes.io/projected/d6fcec3e-1356-443c-90d6-bfd32a029aef-kube-api-access-kcnkm\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:56:40.261947 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.261906 2553 generic.go:356] "Generic (PLEG): container finished" podID="d6fcec3e-1356-443c-90d6-bfd32a029aef" containerID="d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58" exitCode=0 Jul 21 08:56:40.262369 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.261954 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" Jul 21 08:56:40.262369 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.261985 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" event={"ID":"d6fcec3e-1356-443c-90d6-bfd32a029aef","Type":"ContainerDied","Data":"d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58"} Jul 21 08:56:40.262369 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.262036 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b" event={"ID":"d6fcec3e-1356-443c-90d6-bfd32a029aef","Type":"ContainerDied","Data":"5b4148c7232e1a45435f7dc6d6d00f30cc6e2849137603d27d25f83e2ffbb2ac"} Jul 21 08:56:40.262369 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.262068 2553 scope.go:117] "RemoveContainer" containerID="d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58" Jul 21 08:56:40.271444 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.271420 2553 scope.go:117] "RemoveContainer" containerID="d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58" Jul 21 08:56:40.271725 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:56:40.271708 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58\": container with ID starting with d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58 not found: ID does not exist" containerID="d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58" Jul 21 08:56:40.271788 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.271734 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58"} err="failed to get container status \"d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58\": rpc error: code = NotFound desc = could not find container \"d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58\": container with ID starting with d3762f00ecf4f8cfec0dd910f6b17607d6fe045e2fb60a4b4fdb3f87df52ee58 not found: ID does not exist" Jul 21 08:56:40.283915 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.283876 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b"] Jul 21 08:56:40.286316 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.286279 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f491e220-745b6f98f5-gln4b"] Jul 21 08:56:40.674142 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:56:40.674106 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6fcec3e-1356-443c-90d6-bfd32a029aef" path="/var/lib/kubelet/pods/d6fcec3e-1356-443c-90d6-bfd32a029aef/volumes" Jul 21 08:57:34.018780 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:34.018742 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x"] Jul 21 08:57:34.019380 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:34.019021 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" containerID="cri-o://f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29" gracePeriod=30 Jul 21 08:57:39.303080 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.303008 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:57:39.307073 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307049 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-envoy\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307214 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307089 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-socket\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307214 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307136 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istiod-ca-cert\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307214 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307159 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-data\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307219 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x228\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-kube-api-access-9x228\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307252 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-credential-socket\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307293 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-token\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307402 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307377 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-certs\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307601 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307402 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:39.307601 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307416 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-podinfo\") pod \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\" (UID: \"cebfd439-5fda-48f0-b11b-b9b10ab6918d\") " Jul 21 08:57:39.307807 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307773 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 08:57:39.307907 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.307796 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.308250 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.308183 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:39.308250 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.308204 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:39.309172 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.309081 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:39.310217 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.310182 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 08:57:39.311252 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.311231 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 08:57:39.311610 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.311588 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-kube-api-access-9x228" (OuterVolumeSpecName: "kube-api-access-9x228") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "kube-api-access-9x228". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:57:39.311699 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.311677 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cebfd439-5fda-48f0-b11b-b9b10ab6918d" (UID: "cebfd439-5fda-48f0-b11b-b9b10ab6918d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 08:57:39.409163 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409125 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.409163 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409156 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.409163 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409165 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.409436 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409173 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.409436 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409181 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.409436 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409191 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x228\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-kube-api-access-9x228\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.409436 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409200 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cebfd439-5fda-48f0-b11b-b9b10ab6918d-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.409436 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.409208 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cebfd439-5fda-48f0-b11b-b9b10ab6918d-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 08:57:39.565920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.565831 2553 generic.go:356] "Generic (PLEG): container finished" podID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerID="f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29" exitCode=0 Jul 21 08:57:39.565920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.565892 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" Jul 21 08:57:39.565920 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.565915 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" event={"ID":"cebfd439-5fda-48f0-b11b-b9b10ab6918d","Type":"ContainerDied","Data":"f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29"} Jul 21 08:57:39.566177 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.565971 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x" event={"ID":"cebfd439-5fda-48f0-b11b-b9b10ab6918d","Type":"ContainerDied","Data":"c978c32a1848de20fdde4ccc795a57ecab83de2b4d1a2839bd41b6f9f53983b3"} Jul 21 08:57:39.566177 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.565996 2553 scope.go:117] "RemoveContainer" containerID="f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29" Jul 21 08:57:39.575762 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.575737 2553 scope.go:117] "RemoveContainer" containerID="48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce" Jul 21 08:57:39.585116 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.585094 2553 scope.go:117] "RemoveContainer" containerID="f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29" Jul 21 08:57:39.585504 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:57:39.585475 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29\": container with ID starting with f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29 not found: ID does not exist" containerID="f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29" Jul 21 08:57:39.585619 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.585514 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29"} err="failed to get container status \"f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29\": rpc error: code = NotFound desc = could not find container \"f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29\": container with ID starting with f75e2c44e46cc5af58e7fabb3fee85071778931a234fc1346b7f9ebf86ecbd29 not found: ID does not exist" Jul 21 08:57:39.585619 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.585539 2553 scope.go:117] "RemoveContainer" containerID="48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce" Jul 21 08:57:39.585823 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:57:39.585805 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce\": container with ID starting with 48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce not found: ID does not exist" containerID="48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce" Jul 21 08:57:39.585868 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.585828 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce"} err="failed to get container status \"48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce\": rpc error: code = NotFound desc = could not find container \"48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce\": container with ID starting with 48d02ddc9f8b75b1d11e2469d1d47582c80f5df4f446d13a991f5c42e178edce not found: ID does not exist" Jul 21 08:57:39.591441 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.591414 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x"] Jul 21 08:57:39.595097 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:39.595072 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f491e220-openshift-default-595c848747-45g7x"] Jul 21 08:57:40.675233 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:57:40.675196 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" path="/var/lib/kubelet/pods/cebfd439-5fda-48f0-b11b-b9b10ab6918d/volumes" Jul 21 08:58:32.841238 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841205 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b"] Jul 21 08:58:32.841660 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841622 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" Jul 21 08:58:32.841660 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841634 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" Jul 21 08:58:32.841660 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841645 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" Jul 21 08:58:32.841660 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841650 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" Jul 21 08:58:32.841797 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841666 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6fcec3e-1356-443c-90d6-bfd32a029aef" containerName="maas-api" Jul 21 08:58:32.841797 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841673 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6fcec3e-1356-443c-90d6-bfd32a029aef" containerName="maas-api" Jul 21 08:58:32.841797 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841725 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" Jul 21 08:58:32.841797 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.841734 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6fcec3e-1356-443c-90d6-bfd32a029aef" containerName="maas-api" Jul 21 08:58:32.844689 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.844668 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.847663 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.847640 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8d885f-openshift-default-dockercfg-hmxgj\"" Jul 21 08:58:32.930927 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.930891 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b"] Jul 21 08:58:32.933615 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933589 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a853ba-b168-4168-b08f-8945d472a744-istiod-ca-cert\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.933756 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933638 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-socket\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.933756 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933666 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-data\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.933756 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933681 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a853ba-b168-4168-b08f-8945d472a744-istio-podinfo\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.933756 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933710 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-certs\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.934011 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933803 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-credential-socket\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.934011 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933863 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-istio-token\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.934011 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933906 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-envoy\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:32.934011 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:32.933935 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcvbz\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-kube-api-access-tcvbz\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035228 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035178 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-istio-token\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035471 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035244 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-envoy\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035471 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035274 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcvbz\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-kube-api-access-tcvbz\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035471 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035359 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a853ba-b168-4168-b08f-8945d472a744-istiod-ca-cert\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035471 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035403 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-socket\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035471 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035428 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-data\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035471 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035450 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a853ba-b168-4168-b08f-8945d472a744-istio-podinfo\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035793 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035492 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-certs\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035793 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035525 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-credential-socket\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035908 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035820 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-socket\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.035972 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035945 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-data\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.036028 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.035969 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-certs\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.036168 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.036148 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-credential-socket\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.036346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.036262 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a853ba-b168-4168-b08f-8945d472a744-istiod-ca-cert\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.037706 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.037689 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-envoy\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.037990 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.037973 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a853ba-b168-4168-b08f-8945d472a744-istio-podinfo\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.045925 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.045900 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-istio-token\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.046113 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.046093 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcvbz\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-kube-api-access-tcvbz\") pod \"e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.155360 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.155253 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:33.341130 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:58:33.341097 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a853ba_b168_4168_b08f_8945d472a744.slice/crio-2f77de54239dc69ed7b79efa6b1547e7f6719dc3c698c7a3d59a3fd867139b8a WatchSource:0}: Error finding container 2f77de54239dc69ed7b79efa6b1547e7f6719dc3c698c7a3d59a3fd867139b8a: Status 404 returned error can't find the container with id 2f77de54239dc69ed7b79efa6b1547e7f6719dc3c698c7a3d59a3fd867139b8a Jul 21 08:58:33.341486 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.341467 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b"] Jul 21 08:58:33.343163 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.343133 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:58:33.343221 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.343198 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:58:33.343257 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.343226 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:58:33.834035 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.833996 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" event={"ID":"76a853ba-b168-4168-b08f-8945d472a744","Type":"ContainerStarted","Data":"c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861"} Jul 21 08:58:33.834217 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.834054 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" event={"ID":"76a853ba-b168-4168-b08f-8945d472a744","Type":"ContainerStarted","Data":"2f77de54239dc69ed7b79efa6b1547e7f6719dc3c698c7a3d59a3fd867139b8a"} Jul 21 08:58:33.875850 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:33.875798 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" podStartSLOduration=1.875781291 podStartE2EDuration="1.875781291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:58:33.873762037 +0000 UTC m=+2369.759432597" watchObservedRunningTime="2026-07-21 08:58:33.875781291 +0000 UTC m=+2369.761451838" Jul 21 08:58:34.156346 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:34.156225 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:34.161245 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:34.161222 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:34.839587 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:34.839558 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:34.840532 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:34.840510 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:36.314773 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.314718 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr"] Jul 21 08:58:36.315520 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.315498 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebfd439-5fda-48f0-b11b-b9b10ab6918d" containerName="istio-proxy" Jul 21 08:58:36.319139 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.319114 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.326993 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.326973 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8d885f-serving-cert\"" Jul 21 08:58:36.339831 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.339798 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr"] Jul 21 08:58:36.368206 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.368165 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/240b5933-c867-4d74-9705-43a4deb4d72e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.368449 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.368254 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.368449 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.368294 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6ct\" (UniqueName: \"kubernetes.io/projected/240b5933-c867-4d74-9705-43a4deb4d72e-kube-api-access-qb6ct\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.469579 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.469540 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.469787 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.469600 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6ct\" (UniqueName: \"kubernetes.io/projected/240b5933-c867-4d74-9705-43a4deb4d72e-kube-api-access-qb6ct\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.469787 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:58:36.469705 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-serving-cert: secret "maas-api-e2e-inf-a-8d885f-serving-cert" not found Jul 21 08:58:36.469787 ip-10-0-131-134 kubenswrapper[2553]: E0721 08:58:36.469778 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls podName:240b5933-c867-4d74-9705-43a4deb4d72e nodeName:}" failed. No retries permitted until 2026-07-21 08:58:36.969762464 +0000 UTC m=+2372.855432991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls") pod "maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" (UID: "240b5933-c867-4d74-9705-43a4deb4d72e") : secret "maas-api-e2e-inf-a-8d885f-serving-cert" not found Jul 21 08:58:36.469787 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.469711 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/240b5933-c867-4d74-9705-43a4deb4d72e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.470415 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.470393 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/240b5933-c867-4d74-9705-43a4deb4d72e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.492543 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.492510 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6ct\" (UniqueName: \"kubernetes.io/projected/240b5933-c867-4d74-9705-43a4deb4d72e-kube-api-access-qb6ct\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.974520 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.974485 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:36.977016 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:36.976990 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:37.231633 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:37.231542 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:37.414964 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:37.414937 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr"] Jul 21 08:58:37.417115 ip-10-0-131-134 kubenswrapper[2553]: W0721 08:58:37.417083 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240b5933_c867_4d74_9705_43a4deb4d72e.slice/crio-51bb7b4d1c3f9fd1ac04032d51d649496b854d3d06fe488f5eb111fe5ff42ec1 WatchSource:0}: Error finding container 51bb7b4d1c3f9fd1ac04032d51d649496b854d3d06fe488f5eb111fe5ff42ec1: Status 404 returned error can't find the container with id 51bb7b4d1c3f9fd1ac04032d51d649496b854d3d06fe488f5eb111fe5ff42ec1 Jul 21 08:58:37.855469 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:37.855433 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" event={"ID":"240b5933-c867-4d74-9705-43a4deb4d72e","Type":"ContainerStarted","Data":"51bb7b4d1c3f9fd1ac04032d51d649496b854d3d06fe488f5eb111fe5ff42ec1"} Jul 21 08:58:38.862139 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:38.862053 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" event={"ID":"240b5933-c867-4d74-9705-43a4deb4d72e","Type":"ContainerStarted","Data":"b46a9b5dd5d063b8098a922a15cfc2e7bd08cbff2073529644a0e280ccae56e3"} Jul 21 08:58:38.862540 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:38.862148 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:38.903004 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:38.902945 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" podStartSLOduration=1.807100412 podStartE2EDuration="2.902928416s" totalImagesPullingTime="1.095828004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:58:38.899350354 +0000 UTC m=+2374.785020922" watchObservedRunningTime="2026-07-21 08:58:38.902928416 +0000 UTC m=+2374.788598964" Jul 21 08:58:44.873302 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:44.873267 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 08:58:47.223385 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.223342 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8"] Jul 21 08:58:47.227529 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.227510 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.231266 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.231245 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c941d0-serving-cert\"" Jul 21 08:58:47.243465 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.243435 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8"] Jul 21 08:58:47.380238 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.380206 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c171a39f-34a3-4950-8076-656f55f14665-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.380463 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.380272 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c171a39f-34a3-4950-8076-656f55f14665-maas-api-tls\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.380463 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.380384 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5pg\" (UniqueName: \"kubernetes.io/projected/c171a39f-34a3-4950-8076-656f55f14665-kube-api-access-bs5pg\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.481994 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.481909 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c171a39f-34a3-4950-8076-656f55f14665-maas-api-tls\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.481994 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.481965 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5pg\" (UniqueName: \"kubernetes.io/projected/c171a39f-34a3-4950-8076-656f55f14665-kube-api-access-bs5pg\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.482200 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.482013 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c171a39f-34a3-4950-8076-656f55f14665-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.482655 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.482633 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c171a39f-34a3-4950-8076-656f55f14665-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.484780 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.484747 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c171a39f-34a3-4950-8076-656f55f14665-maas-api-tls\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.501004 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.500967 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5pg\" (UniqueName: \"kubernetes.io/projected/c171a39f-34a3-4950-8076-656f55f14665-kube-api-access-bs5pg\") pod \"maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.537764 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.537733 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:47.703665 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.701152 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8"] Jul 21 08:58:47.917068 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.917016 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" event={"ID":"c171a39f-34a3-4950-8076-656f55f14665","Type":"ContainerStarted","Data":"55c6931a7a92ffaf7e8a1414a5042281e5ae109347dc0065dd53c0d185f3b6f1"} Jul 21 08:58:47.919090 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.919058 2553 generic.go:356] "Generic (PLEG): container finished" podID="76a853ba-b168-4168-b08f-8945d472a744" containerID="c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861" exitCode=0 Jul 21 08:58:47.919229 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.919131 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" event={"ID":"76a853ba-b168-4168-b08f-8945d472a744","Type":"ContainerDied","Data":"c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861"} Jul 21 08:58:47.919693 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.919677 2553 scope.go:117] "RemoveContainer" containerID="c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861" Jul 21 08:58:47.920778 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.920750 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:58:47.920917 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.920820 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:58:47.920917 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:47.920850 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 08:58:48.925193 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:48.925158 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" event={"ID":"c171a39f-34a3-4950-8076-656f55f14665","Type":"ContainerStarted","Data":"f0d1ed2d89ed2e900e1337f8a1a701ec62b97118b2934992b4349c921b06a510"} Jul 21 08:58:48.925708 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:48.925255 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:58:48.927088 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:48.927064 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" event={"ID":"76a853ba-b168-4168-b08f-8945d472a744","Type":"ContainerStarted","Data":"c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9"} Jul 21 08:58:48.969443 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:48.969389 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" podStartSLOduration=1.579926889 podStartE2EDuration="1.969372558s" totalImagesPullingTime="389.445669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 08:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 08:58:48.966045082 +0000 UTC m=+2384.851715629" watchObservedRunningTime="2026-07-21 08:58:48.969372558 +0000 UTC m=+2384.855043105" Jul 21 08:58:49.155962 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:49.155925 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:49.157164 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:49.157133 2553 patch_prober.go:28] interesting pod/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 08:58:49.157286 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:49.157193 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 08:58:49.840202 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:49.840164 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:50.156556 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:50.156467 2553 patch_prober.go:28] interesting pod/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 08:58:50.156556 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:50.156530 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 08:58:51.155777 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:51.155735 2553 patch_prober.go:28] interesting pod/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 08:58:51.155990 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:51.155794 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 08:58:52.155765 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:52.155725 2553 patch_prober.go:28] interesting pod/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 08:58:52.156152 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:52.155788 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 08:58:53.155997 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:53.155954 2553 patch_prober.go:28] interesting pod/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 21 08:58:53.156422 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:53.156012 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 21 08:58:54.160350 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:54.160318 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:54.161396 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:54.161374 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 08:58:54.938240 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:58:54.938207 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 08:59:05.146856 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:59:05.146820 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 08:59:05.151401 ip-10-0-131-134 kubenswrapper[2553]: I0721 08:59:05.151380 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 09:00:28.168159 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.168126 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr"] Jul 21 09:00:28.168713 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.168412 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" podUID="240b5933-c867-4d74-9705-43a4deb4d72e" containerName="maas-api" containerID="cri-o://b46a9b5dd5d063b8098a922a15cfc2e7bd08cbff2073529644a0e280ccae56e3" gracePeriod=30 Jul 21 09:00:28.434213 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.434125 2553 generic.go:356] "Generic (PLEG): container finished" podID="240b5933-c867-4d74-9705-43a4deb4d72e" containerID="b46a9b5dd5d063b8098a922a15cfc2e7bd08cbff2073529644a0e280ccae56e3" exitCode=0 Jul 21 09:00:28.434389 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.434200 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" event={"ID":"240b5933-c867-4d74-9705-43a4deb4d72e","Type":"ContainerDied","Data":"b46a9b5dd5d063b8098a922a15cfc2e7bd08cbff2073529644a0e280ccae56e3"} Jul 21 09:00:28.434389 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.434272 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" event={"ID":"240b5933-c867-4d74-9705-43a4deb4d72e","Type":"ContainerDied","Data":"51bb7b4d1c3f9fd1ac04032d51d649496b854d3d06fe488f5eb111fe5ff42ec1"} Jul 21 09:00:28.434389 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.434288 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51bb7b4d1c3f9fd1ac04032d51d649496b854d3d06fe488f5eb111fe5ff42ec1" Jul 21 09:00:28.435302 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.435281 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 09:00:28.589858 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.589817 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/240b5933-c867-4d74-9705-43a4deb4d72e-openshift-service-ca\") pod \"240b5933-c867-4d74-9705-43a4deb4d72e\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " Jul 21 09:00:28.589858 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.589859 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls\") pod \"240b5933-c867-4d74-9705-43a4deb4d72e\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " Jul 21 09:00:28.590117 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.589933 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb6ct\" (UniqueName: \"kubernetes.io/projected/240b5933-c867-4d74-9705-43a4deb4d72e-kube-api-access-qb6ct\") pod \"240b5933-c867-4d74-9705-43a4deb4d72e\" (UID: \"240b5933-c867-4d74-9705-43a4deb4d72e\") " Jul 21 09:00:28.590212 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.590176 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240b5933-c867-4d74-9705-43a4deb4d72e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "240b5933-c867-4d74-9705-43a4deb4d72e" (UID: "240b5933-c867-4d74-9705-43a4deb4d72e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:00:28.590368 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.590352 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/240b5933-c867-4d74-9705-43a4deb4d72e-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:28.592094 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.592071 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240b5933-c867-4d74-9705-43a4deb4d72e-kube-api-access-qb6ct" (OuterVolumeSpecName: "kube-api-access-qb6ct") pod "240b5933-c867-4d74-9705-43a4deb4d72e" (UID: "240b5933-c867-4d74-9705-43a4deb4d72e"). InnerVolumeSpecName "kube-api-access-qb6ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:00:28.592094 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.592086 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "240b5933-c867-4d74-9705-43a4deb4d72e" (UID: "240b5933-c867-4d74-9705-43a4deb4d72e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:00:28.691422 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.691354 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb6ct\" (UniqueName: \"kubernetes.io/projected/240b5933-c867-4d74-9705-43a4deb4d72e-kube-api-access-qb6ct\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:28.691422 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:28.691379 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/240b5933-c867-4d74-9705-43a4deb4d72e-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:29.439351 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:29.439292 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr" Jul 21 09:00:29.456761 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:29.456724 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr"] Jul 21 09:00:29.464553 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:29.464524 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d885f-7ffdc94cb5-vbjjr"] Jul 21 09:00:30.674627 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:30.674592 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="240b5933-c867-4d74-9705-43a4deb4d72e" path="/var/lib/kubelet/pods/240b5933-c867-4d74-9705-43a4deb4d72e/volumes" Jul 21 09:00:39.489302 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:39.489266 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b"] Jul 21 09:00:39.489926 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:39.489516 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" containerID="cri-o://c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9" gracePeriod=30 Jul 21 09:00:44.756542 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.756476 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 09:00:44.851586 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851554 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a853ba-b168-4168-b08f-8945d472a744-istiod-ca-cert\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851586 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851595 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-envoy\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851628 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcvbz\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-kube-api-access-tcvbz\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851645 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-data\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851712 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a853ba-b168-4168-b08f-8945d472a744-istio-podinfo\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851751 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-credential-socket\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851775 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-socket\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851798 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-certs\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.851851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.851832 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-istio-token\") pod \"76a853ba-b168-4168-b08f-8945d472a744\" (UID: \"76a853ba-b168-4168-b08f-8945d472a744\") " Jul 21 09:00:44.852211 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852088 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:44.852211 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852111 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a853ba-b168-4168-b08f-8945d472a744-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:00:44.852351 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852300 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:44.852617 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852595 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.852675 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852623 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.852675 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852637 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a853ba-b168-4168-b08f-8945d472a744-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.852675 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852645 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:44.852799 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.852720 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-data" (OuterVolumeSpecName: "istio-data") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:44.854108 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.854075 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76a853ba-b168-4168-b08f-8945d472a744-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 09:00:44.854108 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.854083 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-kube-api-access-tcvbz" (OuterVolumeSpecName: "kube-api-access-tcvbz") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "kube-api-access-tcvbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:00:44.854266 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.854143 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:00:44.854494 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.854475 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-istio-token" (OuterVolumeSpecName: "istio-token") pod "76a853ba-b168-4168-b08f-8945d472a744" (UID: "76a853ba-b168-4168-b08f-8945d472a744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:00:44.953082 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.953046 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.953082 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.953075 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcvbz\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-kube-api-access-tcvbz\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.953082 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.953085 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.953082 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.953094 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a853ba-b168-4168-b08f-8945d472a744-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.953371 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.953102 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a853ba-b168-4168-b08f-8945d472a744-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:44.953371 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:44.953110 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a853ba-b168-4168-b08f-8945d472a744-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:45.523667 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.523631 2553 generic.go:356] "Generic (PLEG): container finished" podID="76a853ba-b168-4168-b08f-8945d472a744" containerID="c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9" exitCode=0 Jul 21 09:00:45.523842 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.523684 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" Jul 21 09:00:45.523842 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.523692 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" event={"ID":"76a853ba-b168-4168-b08f-8945d472a744","Type":"ContainerDied","Data":"c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9"} Jul 21 09:00:45.523842 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.523749 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b" event={"ID":"76a853ba-b168-4168-b08f-8945d472a744","Type":"ContainerDied","Data":"2f77de54239dc69ed7b79efa6b1547e7f6719dc3c698c7a3d59a3fd867139b8a"} Jul 21 09:00:45.523842 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.523779 2553 scope.go:117] "RemoveContainer" containerID="c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9" Jul 21 09:00:45.533296 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.533277 2553 scope.go:117] "RemoveContainer" containerID="c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861" Jul 21 09:00:45.542075 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.542051 2553 scope.go:117] "RemoveContainer" containerID="c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9" Jul 21 09:00:45.542368 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:00:45.542345 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9\": container with ID starting with c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9 not found: ID does not exist" containerID="c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9" Jul 21 09:00:45.542433 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.542379 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9"} err="failed to get container status \"c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9\": rpc error: code = NotFound desc = could not find container \"c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9\": container with ID starting with c3255a210e0fbdfabdf5751ee1cdefb7c1caf5424ee81b54c85b5b8b669f3fb9 not found: ID does not exist" Jul 21 09:00:45.542433 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.542399 2553 scope.go:117] "RemoveContainer" containerID="c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861" Jul 21 09:00:45.542663 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:00:45.542646 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861\": container with ID starting with c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861 not found: ID does not exist" containerID="c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861" Jul 21 09:00:45.542717 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.542665 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861"} err="failed to get container status \"c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861\": rpc error: code = NotFound desc = could not find container \"c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861\": container with ID starting with c47a333a8553c59b0ca4f0a627cb1f26587c69599ecb6bf52f86e786c48f8861 not found: ID does not exist" Jul 21 09:00:45.546726 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.546698 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b"] Jul 21 09:00:45.555560 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:45.555539 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8d885f-openshift-default-5b6f659799-hld7b"] Jul 21 09:00:46.673887 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:46.673850 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a853ba-b168-4168-b08f-8945d472a744" path="/var/lib/kubelet/pods/76a853ba-b168-4168-b08f-8945d472a744/volumes" Jul 21 09:00:50.419910 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.419869 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8"] Jul 21 09:00:50.420435 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.420180 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" podUID="c171a39f-34a3-4950-8076-656f55f14665" containerName="maas-api" containerID="cri-o://f0d1ed2d89ed2e900e1337f8a1a701ec62b97118b2934992b4349c921b06a510" gracePeriod=30 Jul 21 09:00:50.551528 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.551489 2553 generic.go:356] "Generic (PLEG): container finished" podID="c171a39f-34a3-4950-8076-656f55f14665" containerID="f0d1ed2d89ed2e900e1337f8a1a701ec62b97118b2934992b4349c921b06a510" exitCode=0 Jul 21 09:00:50.551689 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.551556 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" event={"ID":"c171a39f-34a3-4950-8076-656f55f14665","Type":"ContainerDied","Data":"f0d1ed2d89ed2e900e1337f8a1a701ec62b97118b2934992b4349c921b06a510"} Jul 21 09:00:50.683402 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.683336 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 09:00:50.710213 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.710175 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c171a39f-34a3-4950-8076-656f55f14665-maas-api-tls\") pod \"c171a39f-34a3-4950-8076-656f55f14665\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " Jul 21 09:00:50.710419 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.710302 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs5pg\" (UniqueName: \"kubernetes.io/projected/c171a39f-34a3-4950-8076-656f55f14665-kube-api-access-bs5pg\") pod \"c171a39f-34a3-4950-8076-656f55f14665\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " Jul 21 09:00:50.710505 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.710421 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c171a39f-34a3-4950-8076-656f55f14665-openshift-service-ca\") pod \"c171a39f-34a3-4950-8076-656f55f14665\" (UID: \"c171a39f-34a3-4950-8076-656f55f14665\") " Jul 21 09:00:50.710832 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.710804 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c171a39f-34a3-4950-8076-656f55f14665-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c171a39f-34a3-4950-8076-656f55f14665" (UID: "c171a39f-34a3-4950-8076-656f55f14665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:00:50.712507 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.712478 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c171a39f-34a3-4950-8076-656f55f14665-kube-api-access-bs5pg" (OuterVolumeSpecName: "kube-api-access-bs5pg") pod "c171a39f-34a3-4950-8076-656f55f14665" (UID: "c171a39f-34a3-4950-8076-656f55f14665"). InnerVolumeSpecName "kube-api-access-bs5pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:00:50.712598 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.712478 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c171a39f-34a3-4950-8076-656f55f14665-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c171a39f-34a3-4950-8076-656f55f14665" (UID: "c171a39f-34a3-4950-8076-656f55f14665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:00:50.811977 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.811942 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs5pg\" (UniqueName: \"kubernetes.io/projected/c171a39f-34a3-4950-8076-656f55f14665-kube-api-access-bs5pg\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:50.811977 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.811971 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c171a39f-34a3-4950-8076-656f55f14665-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:50.811977 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:50.811982 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c171a39f-34a3-4950-8076-656f55f14665-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:00:51.557231 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:51.557196 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" Jul 21 09:00:51.557650 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:51.557230 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8" event={"ID":"c171a39f-34a3-4950-8076-656f55f14665","Type":"ContainerDied","Data":"55c6931a7a92ffaf7e8a1414a5042281e5ae109347dc0065dd53c0d185f3b6f1"} Jul 21 09:00:51.557650 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:51.557297 2553 scope.go:117] "RemoveContainer" containerID="f0d1ed2d89ed2e900e1337f8a1a701ec62b97118b2934992b4349c921b06a510" Jul 21 09:00:51.580762 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:51.580733 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8"] Jul 21 09:00:51.582646 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:51.582625 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c941d0-5ddfb5bcf4-tmnl8"] Jul 21 09:00:52.674138 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:00:52.674108 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c171a39f-34a3-4950-8076-656f55f14665" path="/var/lib/kubelet/pods/c171a39f-34a3-4950-8076-656f55f14665/volumes" Jul 21 09:01:02.965229 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965151 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh"] Jul 21 09:01:02.965644 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965622 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" Jul 21 09:01:02.965644 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965636 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" Jul 21 09:01:02.965710 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965647 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c171a39f-34a3-4950-8076-656f55f14665" containerName="maas-api" Jul 21 09:01:02.965710 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965652 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c171a39f-34a3-4950-8076-656f55f14665" containerName="maas-api" Jul 21 09:01:02.965710 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965676 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="240b5933-c867-4d74-9705-43a4deb4d72e" containerName="maas-api" Jul 21 09:01:02.965710 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965682 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="240b5933-c867-4d74-9705-43a4deb4d72e" containerName="maas-api" Jul 21 09:01:02.965832 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965741 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" Jul 21 09:01:02.965832 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965748 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" Jul 21 09:01:02.965832 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965754 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="240b5933-c867-4d74-9705-43a4deb4d72e" containerName="maas-api" Jul 21 09:01:02.965832 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.965761 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c171a39f-34a3-4950-8076-656f55f14665" containerName="maas-api" Jul 21 09:01:02.970326 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.970290 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:02.974025 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.974000 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b88480-openshift-default-dockercfg-wl7kv\"" Jul 21 09:01:02.995265 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:02.995233 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh"] Jul 21 09:01:03.021427 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021390 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-certs\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021454 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-socket\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021496 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-data\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021514 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-credential-socket\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021536 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2d4j\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-kube-api-access-g2d4j\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-envoy\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021615 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-podinfo\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021629 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-token\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.021796 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.021645 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istiod-ca-cert\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122455 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122423 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-socket\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122455 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122469 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-data\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122685 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122489 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-credential-socket\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122685 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122516 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2d4j\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-kube-api-access-g2d4j\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122685 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122546 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-envoy\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122685 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-podinfo\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122685 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122604 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-token\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122685 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122626 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istiod-ca-cert\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122946 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122695 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-certs\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122946 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122887 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-credential-socket\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.122946 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122928 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-socket\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.123093 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.122982 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-data\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.123093 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.123024 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-certs\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.123556 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.123532 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istiod-ca-cert\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.125166 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.125147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-podinfo\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.125283 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.125223 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-envoy\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.141197 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.141169 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-token\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.141354 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.141227 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2d4j\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-kube-api-access-g2d4j\") pod \"e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.280686 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.280599 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:03.489990 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.489931 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh"] Jul 21 09:01:03.492452 ip-10-0-131-134 kubenswrapper[2553]: W0721 09:01:03.492424 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39908d13_31e9_44ad_91c1_7e45a1ff32d5.slice/crio-7178148f4fee03692d2996245d8cda59638e0ea32c4061e4df6ddd10bd25c1cb WatchSource:0}: Error finding container 7178148f4fee03692d2996245d8cda59638e0ea32c4061e4df6ddd10bd25c1cb: Status 404 returned error can't find the container with id 7178148f4fee03692d2996245d8cda59638e0ea32c4061e4df6ddd10bd25c1cb Jul 21 09:01:03.494548 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.494512 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:01:03.494654 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.494604 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:01:03.494718 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.494651 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:01:03.628110 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.628024 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" event={"ID":"39908d13-31e9-44ad-91c1-7e45a1ff32d5","Type":"ContainerStarted","Data":"63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708"} Jul 21 09:01:03.628110 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.628084 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" event={"ID":"39908d13-31e9-44ad-91c1-7e45a1ff32d5","Type":"ContainerStarted","Data":"7178148f4fee03692d2996245d8cda59638e0ea32c4061e4df6ddd10bd25c1cb"} Jul 21 09:01:03.697504 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:03.697443 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podStartSLOduration=1.697419963 podStartE2EDuration="1.697419963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:01:03.696237849 +0000 UTC m=+2519.581908409" watchObservedRunningTime="2026-07-21 09:01:03.697419963 +0000 UTC m=+2519.583090515" Jul 21 09:01:04.280777 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:04.280744 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:04.286268 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:04.286238 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:04.632767 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:04.632690 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:04.633576 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:04.633556 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:06.020941 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.020901 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj"] Jul 21 09:01:06.021360 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.021329 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" Jul 21 09:01:06.021360 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.021341 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a853ba-b168-4168-b08f-8945d472a744" containerName="istio-proxy" Jul 21 09:01:06.025757 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.025729 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.032519 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.032483 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b88480-serving-cert\"" Jul 21 09:01:06.079693 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.079652 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj"] Jul 21 09:01:06.150868 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.150834 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2qr\" (UniqueName: \"kubernetes.io/projected/f609a355-baa0-407e-84fa-a368d152f6ae-kube-api-access-sj2qr\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.151042 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.150876 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609a355-baa0-407e-84fa-a368d152f6ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.151042 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.150947 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.252326 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.252271 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2qr\" (UniqueName: \"kubernetes.io/projected/f609a355-baa0-407e-84fa-a368d152f6ae-kube-api-access-sj2qr\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.252497 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.252336 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609a355-baa0-407e-84fa-a368d152f6ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.252497 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.252375 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.252637 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:01:06.252510 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-serving-cert: secret "maas-api-e2e-api-a-b88480-serving-cert" not found Jul 21 09:01:06.252637 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:01:06.252585 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls podName:f609a355-baa0-407e-84fa-a368d152f6ae nodeName:}" failed. No retries permitted until 2026-07-21 09:01:06.752564589 +0000 UTC m=+2522.638235120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls") pod "maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" (UID: "f609a355-baa0-407e-84fa-a368d152f6ae") : secret "maas-api-e2e-api-a-b88480-serving-cert" not found Jul 21 09:01:06.253058 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.253035 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609a355-baa0-407e-84fa-a368d152f6ae-openshift-service-ca\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.262976 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.262944 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2qr\" (UniqueName: \"kubernetes.io/projected/f609a355-baa0-407e-84fa-a368d152f6ae-kube-api-access-sj2qr\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.758512 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.758474 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.761024 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.760999 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls\") pod \"maas-api-e2e-api-a-b88480-5948b79cc-gc5tj\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:06.937386 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:06.937347 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:07.089095 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:07.088952 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj"] Jul 21 09:01:07.091513 ip-10-0-131-134 kubenswrapper[2553]: W0721 09:01:07.091488 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf609a355_baa0_407e_84fa_a368d152f6ae.slice/crio-0c084f9362d687419522619de0dfa0ecd2819905a40cce271cf4448ba2068ae4 WatchSource:0}: Error finding container 0c084f9362d687419522619de0dfa0ecd2819905a40cce271cf4448ba2068ae4: Status 404 returned error can't find the container with id 0c084f9362d687419522619de0dfa0ecd2819905a40cce271cf4448ba2068ae4 Jul 21 09:01:07.092941 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:07.092922 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 09:01:07.649296 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:07.649271 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" event={"ID":"f609a355-baa0-407e-84fa-a368d152f6ae","Type":"ContainerStarted","Data":"0c084f9362d687419522619de0dfa0ecd2819905a40cce271cf4448ba2068ae4"} Jul 21 09:01:08.656662 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:08.656626 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" event={"ID":"f609a355-baa0-407e-84fa-a368d152f6ae","Type":"ContainerStarted","Data":"8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a"} Jul 21 09:01:08.657073 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:08.656749 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:08.674496 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:08.674443 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" podStartSLOduration=3.121344681 podStartE2EDuration="3.674424627s" totalImagesPullingTime="553.079946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:01:08.67432599 +0000 UTC m=+2524.559996529" watchObservedRunningTime="2026-07-21 09:01:08.674424627 +0000 UTC m=+2524.560095178" Jul 21 09:01:13.948342 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:13.948122 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87"] Jul 21 09:01:13.956786 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:13.956755 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:13.962837 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:13.962735 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bb8bc8-serving-cert\"" Jul 21 09:01:13.964632 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:13.963767 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87"] Jul 21 09:01:14.031005 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.030965 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-openshift-service-ca\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.031189 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.031088 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47mr\" (UniqueName: \"kubernetes.io/projected/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-kube-api-access-f47mr\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.031189 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.031150 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-maas-api-tls\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.132411 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.132368 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-openshift-service-ca\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.132574 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.132424 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47mr\" (UniqueName: \"kubernetes.io/projected/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-kube-api-access-f47mr\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.132574 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.132478 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-maas-api-tls\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.133133 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.133096 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-openshift-service-ca\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.135115 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.135089 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-maas-api-tls\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.150546 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.150508 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47mr\" (UniqueName: \"kubernetes.io/projected/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-kube-api-access-f47mr\") pod \"maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.281618 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.281512 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:14.536899 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.536873 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87"] Jul 21 09:01:14.539637 ip-10-0-131-134 kubenswrapper[2553]: W0721 09:01:14.539598 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7dfac4_5dd2_4fa5_8523_979c65ff7784.slice/crio-bf723b40c22a5de1a358306c76545f03b1349ca8ed286330334c62aee490336b WatchSource:0}: Error finding container bf723b40c22a5de1a358306c76545f03b1349ca8ed286330334c62aee490336b: Status 404 returned error can't find the container with id bf723b40c22a5de1a358306c76545f03b1349ca8ed286330334c62aee490336b Jul 21 09:01:14.675584 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.675559 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:14.713532 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:14.713485 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" event={"ID":"3a7dfac4-5dd2-4fa5-8523-979c65ff7784","Type":"ContainerStarted","Data":"bf723b40c22a5de1a358306c76545f03b1349ca8ed286330334c62aee490336b"} Jul 21 09:01:15.721224 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:15.721176 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" event={"ID":"3a7dfac4-5dd2-4fa5-8523-979c65ff7784","Type":"ContainerStarted","Data":"dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0"} Jul 21 09:01:15.721794 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:15.721291 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:15.741933 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:15.741880 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" podStartSLOduration=2.376889903 podStartE2EDuration="2.741861816s" totalImagesPullingTime="364.971913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:01:15.741161427 +0000 UTC m=+2531.626831976" watchObservedRunningTime="2026-07-21 09:01:15.741861816 +0000 UTC m=+2531.627532404" Jul 21 09:01:20.674523 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:20.674475 2553 patch_prober.go:28] interesting pod/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 09:01:20.675054 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:20.674533 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 09:01:21.732155 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:21.732128 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:01:21.764339 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:21.764086 2553 generic.go:356] "Generic (PLEG): container finished" podID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerID="63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708" exitCode=0 Jul 21 09:01:21.764513 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:21.764260 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" event={"ID":"39908d13-31e9-44ad-91c1-7e45a1ff32d5","Type":"ContainerDied","Data":"63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708"} Jul 21 09:01:21.764935 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:21.764915 2553 scope.go:117] "RemoveContainer" containerID="63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708" Jul 21 09:01:21.766186 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:21.766146 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:01:21.766332 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:21.766236 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:01:21.766332 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:21.766284 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:01:22.771831 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:22.771795 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" event={"ID":"39908d13-31e9-44ad-91c1-7e45a1ff32d5","Type":"ContainerStarted","Data":"b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b"} Jul 21 09:01:23.281022 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:23.280984 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:23.281202 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:23.281169 2553 patch_prober.go:28] interesting pod/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 21 09:01:23.281250 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:23.281214 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 21 09:01:24.281387 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:24.281350 2553 patch_prober.go:28] interesting pod/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 21 09:01:24.281783 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:24.281405 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 21 09:01:25.281598 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:25.281567 2553 patch_prober.go:28] interesting pod/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 21 09:01:25.281991 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:25.281616 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 21 09:01:26.281463 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:26.281429 2553 patch_prober.go:28] interesting pod/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 21 09:01:26.281648 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:26.281486 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 21 09:01:27.284535 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:27.284494 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:27.285013 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:27.284911 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:27.285746 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:27.285723 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:01:43.216076 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.216042 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj"] Jul 21 09:01:43.216507 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.216254 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" podUID="f609a355-baa0-407e-84fa-a368d152f6ae" containerName="maas-api" containerID="cri-o://8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a" gracePeriod=30 Jul 21 09:01:43.478144 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.478072 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:43.523916 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.523879 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj2qr\" (UniqueName: \"kubernetes.io/projected/f609a355-baa0-407e-84fa-a368d152f6ae-kube-api-access-sj2qr\") pod \"f609a355-baa0-407e-84fa-a368d152f6ae\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " Jul 21 09:01:43.524103 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.523989 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls\") pod \"f609a355-baa0-407e-84fa-a368d152f6ae\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " Jul 21 09:01:43.524103 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.524050 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609a355-baa0-407e-84fa-a368d152f6ae-openshift-service-ca\") pod \"f609a355-baa0-407e-84fa-a368d152f6ae\" (UID: \"f609a355-baa0-407e-84fa-a368d152f6ae\") " Jul 21 09:01:43.524481 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.524449 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f609a355-baa0-407e-84fa-a368d152f6ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f609a355-baa0-407e-84fa-a368d152f6ae" (UID: "f609a355-baa0-407e-84fa-a368d152f6ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:01:43.526202 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.526159 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f609a355-baa0-407e-84fa-a368d152f6ae-kube-api-access-sj2qr" (OuterVolumeSpecName: "kube-api-access-sj2qr") pod "f609a355-baa0-407e-84fa-a368d152f6ae" (UID: "f609a355-baa0-407e-84fa-a368d152f6ae"). InnerVolumeSpecName "kube-api-access-sj2qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:01:43.526348 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.526218 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f609a355-baa0-407e-84fa-a368d152f6ae" (UID: "f609a355-baa0-407e-84fa-a368d152f6ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:01:43.625647 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.625607 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f609a355-baa0-407e-84fa-a368d152f6ae-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:01:43.625647 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.625641 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f609a355-baa0-407e-84fa-a368d152f6ae-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:01:43.625647 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.625651 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj2qr\" (UniqueName: \"kubernetes.io/projected/f609a355-baa0-407e-84fa-a368d152f6ae-kube-api-access-sj2qr\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:01:43.889873 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.889788 2553 generic.go:356] "Generic (PLEG): container finished" podID="f609a355-baa0-407e-84fa-a368d152f6ae" containerID="8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a" exitCode=0 Jul 21 09:01:43.889873 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.889837 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" Jul 21 09:01:43.889873 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.889854 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" event={"ID":"f609a355-baa0-407e-84fa-a368d152f6ae","Type":"ContainerDied","Data":"8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a"} Jul 21 09:01:43.890124 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.889894 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj" event={"ID":"f609a355-baa0-407e-84fa-a368d152f6ae","Type":"ContainerDied","Data":"0c084f9362d687419522619de0dfa0ecd2819905a40cce271cf4448ba2068ae4"} Jul 21 09:01:43.890124 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.889915 2553 scope.go:117] "RemoveContainer" containerID="8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a" Jul 21 09:01:43.899570 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.899553 2553 scope.go:117] "RemoveContainer" containerID="8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a" Jul 21 09:01:43.899850 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:01:43.899833 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a\": container with ID starting with 8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a not found: ID does not exist" containerID="8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a" Jul 21 09:01:43.899928 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.899857 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a"} err="failed to get container status \"8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a\": rpc error: code = NotFound desc = could not find container \"8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a\": container with ID starting with 8f9bb444eaa1ed3700b383ca893487cf95ac60658d8d1ec3fd41ff73064b481a not found: ID does not exist" Jul 21 09:01:43.912585 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.912547 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj"] Jul 21 09:01:43.914410 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:43.914389 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b88480-5948b79cc-gc5tj"] Jul 21 09:01:44.674223 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:44.674189 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f609a355-baa0-407e-84fa-a368d152f6ae" path="/var/lib/kubelet/pods/f609a355-baa0-407e-84fa-a368d152f6ae/volumes" Jul 21 09:01:55.170303 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:55.170265 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh"] Jul 21 09:01:55.170808 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:55.170600 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" containerID="cri-o://b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b" gracePeriod=30 Jul 21 09:01:57.287758 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:57.287713 2553 patch_prober.go:28] interesting pod/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 09:01:57.288139 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:01:57.287792 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 09:02:00.437434 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.437372 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:02:00.487866 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.487832 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-podinfo\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.487866 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.487880 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2d4j\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-kube-api-access-g2d4j\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488126 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.487902 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istiod-ca-cert\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488126 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.487939 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-token\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488126 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.487977 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-credential-socket\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488126 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488014 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-certs\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488126 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488059 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-envoy\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488444 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488140 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-data\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488444 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488192 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-socket\") pod \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\" (UID: \"39908d13-31e9-44ad-91c1-7e45a1ff32d5\") " Jul 21 09:02:00.488444 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488304 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:02:00.488621 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488436 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:02:00.488621 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488555 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:02:00.488879 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488830 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.488879 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488754 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:02:00.488879 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488857 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.489148 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.488892 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.489727 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.489694 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:02:00.490292 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.490271 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 09:02:00.490863 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.490833 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:02:00.490969 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.490918 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-kube-api-access-g2d4j" (OuterVolumeSpecName: "kube-api-access-g2d4j") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "kube-api-access-g2d4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:02:00.490969 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.490934 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "39908d13-31e9-44ad-91c1-7e45a1ff32d5" (UID: "39908d13-31e9-44ad-91c1-7e45a1ff32d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:02:00.589678 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.589641 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.589678 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.589671 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2d4j\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-kube-api-access-g2d4j\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.589678 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.589682 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.589678 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.589690 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.589953 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.589698 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.589953 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.589706 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39908d13-31e9-44ad-91c1-7e45a1ff32d5-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:00.976218 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.976175 2553 generic.go:356] "Generic (PLEG): container finished" podID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerID="b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b" exitCode=0 Jul 21 09:02:00.976425 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.976233 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" Jul 21 09:02:00.976425 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.976261 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" event={"ID":"39908d13-31e9-44ad-91c1-7e45a1ff32d5","Type":"ContainerDied","Data":"b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b"} Jul 21 09:02:00.976425 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.976342 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh" event={"ID":"39908d13-31e9-44ad-91c1-7e45a1ff32d5","Type":"ContainerDied","Data":"7178148f4fee03692d2996245d8cda59638e0ea32c4061e4df6ddd10bd25c1cb"} Jul 21 09:02:00.976425 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.976369 2553 scope.go:117] "RemoveContainer" containerID="b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b" Jul 21 09:02:00.985948 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.985931 2553 scope.go:117] "RemoveContainer" containerID="63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708" Jul 21 09:02:00.994428 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.994410 2553 scope.go:117] "RemoveContainer" containerID="b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b" Jul 21 09:02:00.994699 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:02:00.994678 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b\": container with ID starting with b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b not found: ID does not exist" containerID="b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b" Jul 21 09:02:00.994771 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.994707 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b"} err="failed to get container status \"b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b\": rpc error: code = NotFound desc = could not find container \"b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b\": container with ID starting with b236a7b3622f48bedfcba3f98b9e95859696456a3613a7fd1c491530a228626b not found: ID does not exist" Jul 21 09:02:00.994771 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.994728 2553 scope.go:117] "RemoveContainer" containerID="63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708" Jul 21 09:02:00.994982 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:02:00.994960 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708\": container with ID starting with 63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708 not found: ID does not exist" containerID="63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708" Jul 21 09:02:00.995044 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.994993 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708"} err="failed to get container status \"63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708\": rpc error: code = NotFound desc = could not find container \"63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708\": container with ID starting with 63478a0fa5cebbfc538eaad44e0eaafecd73911a93bcc65276f67dc1ddc12708 not found: ID does not exist" Jul 21 09:02:00.995130 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:00.995112 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh"] Jul 21 09:02:01.000519 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:01.000500 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b88480-openshift-default-6d48ddc8bc-5psjh"] Jul 21 09:02:02.674674 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:02.674642 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" path="/var/lib/kubelet/pods/39908d13-31e9-44ad-91c1-7e45a1ff32d5/volumes" Jul 21 09:02:11.267101 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.267063 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87"] Jul 21 09:02:11.267573 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.267340 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" podUID="3a7dfac4-5dd2-4fa5-8523-979c65ff7784" containerName="maas-api" containerID="cri-o://dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0" gracePeriod=30 Jul 21 09:02:11.527769 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.527704 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:02:11.700473 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.700431 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-openshift-service-ca\") pod \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " Jul 21 09:02:11.700652 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.700485 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f47mr\" (UniqueName: \"kubernetes.io/projected/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-kube-api-access-f47mr\") pod \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " Jul 21 09:02:11.700652 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.700557 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-maas-api-tls\") pod \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\" (UID: \"3a7dfac4-5dd2-4fa5-8523-979c65ff7784\") " Jul 21 09:02:11.700896 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.700868 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a7dfac4-5dd2-4fa5-8523-979c65ff7784" (UID: "3a7dfac4-5dd2-4fa5-8523-979c65ff7784"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:02:11.702818 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.702788 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a7dfac4-5dd2-4fa5-8523-979c65ff7784" (UID: "3a7dfac4-5dd2-4fa5-8523-979c65ff7784"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:02:11.702920 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.702871 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-kube-api-access-f47mr" (OuterVolumeSpecName: "kube-api-access-f47mr") pod "3a7dfac4-5dd2-4fa5-8523-979c65ff7784" (UID: "3a7dfac4-5dd2-4fa5-8523-979c65ff7784"). InnerVolumeSpecName "kube-api-access-f47mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:02:11.801771 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.801686 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:11.801771 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.801718 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f47mr\" (UniqueName: \"kubernetes.io/projected/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-kube-api-access-f47mr\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:11.801771 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:11.801729 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a7dfac4-5dd2-4fa5-8523-979c65ff7784-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:02:12.035528 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.035496 2553 generic.go:356] "Generic (PLEG): container finished" podID="3a7dfac4-5dd2-4fa5-8523-979c65ff7784" containerID="dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0" exitCode=0 Jul 21 09:02:12.035709 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.035541 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" Jul 21 09:02:12.035709 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.035586 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" event={"ID":"3a7dfac4-5dd2-4fa5-8523-979c65ff7784","Type":"ContainerDied","Data":"dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0"} Jul 21 09:02:12.035709 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.035637 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87" event={"ID":"3a7dfac4-5dd2-4fa5-8523-979c65ff7784","Type":"ContainerDied","Data":"bf723b40c22a5de1a358306c76545f03b1349ca8ed286330334c62aee490336b"} Jul 21 09:02:12.035709 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.035659 2553 scope.go:117] "RemoveContainer" containerID="dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0" Jul 21 09:02:12.044756 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.044737 2553 scope.go:117] "RemoveContainer" containerID="dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0" Jul 21 09:02:12.045049 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:02:12.045029 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0\": container with ID starting with dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0 not found: ID does not exist" containerID="dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0" Jul 21 09:02:12.045088 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.045060 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0"} err="failed to get container status \"dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0\": rpc error: code = NotFound desc = could not find container \"dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0\": container with ID starting with dae13507ba6f37cf63534df4e508633b3282d018b50facf23ba9eaf2ff6b8fe0 not found: ID does not exist" Jul 21 09:02:12.063475 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.063399 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87"] Jul 21 09:02:12.072402 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.072372 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bb8bc8-d9c7657fc-b4h87"] Jul 21 09:02:12.674675 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:12.674639 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a7dfac4-5dd2-4fa5-8523-979c65ff7784" path="/var/lib/kubelet/pods/3a7dfac4-5dd2-4fa5-8523-979c65ff7784/volumes" Jul 21 09:02:23.667570 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.667533 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx"] Jul 21 09:02:23.668156 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668134 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f609a355-baa0-407e-84fa-a368d152f6ae" containerName="maas-api" Jul 21 09:02:23.668258 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668158 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f609a355-baa0-407e-84fa-a368d152f6ae" containerName="maas-api" Jul 21 09:02:23.668258 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668178 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" Jul 21 09:02:23.668258 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668186 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" Jul 21 09:02:23.668258 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668199 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7dfac4-5dd2-4fa5-8523-979c65ff7784" containerName="maas-api" Jul 21 09:02:23.668258 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668207 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7dfac4-5dd2-4fa5-8523-979c65ff7784" containerName="maas-api" Jul 21 09:02:23.668258 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668223 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" Jul 21 09:02:23.668258 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668231 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" Jul 21 09:02:23.668623 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668357 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7dfac4-5dd2-4fa5-8523-979c65ff7784" containerName="maas-api" Jul 21 09:02:23.668623 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668373 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f609a355-baa0-407e-84fa-a368d152f6ae" containerName="maas-api" Jul 21 09:02:23.668623 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668386 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" Jul 21 09:02:23.668623 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.668396 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="39908d13-31e9-44ad-91c1-7e45a1ff32d5" containerName="istio-proxy" Jul 21 09:02:23.671639 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.671619 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.675890 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.675871 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-824408-openshift-default-dockercfg-v9ndz\"" Jul 21 09:02:23.694528 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.694496 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx"] Jul 21 09:02:23.711654 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a742c7-653c-43a5-8f0a-268d7cdb5526-istiod-ca-cert\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.711789 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711663 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-envoy\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.711789 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711722 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-certs\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.711789 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-token\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.711933 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711825 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-credential-socket\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.711933 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711863 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-data\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.711933 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711881 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-socket\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.711933 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711898 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-podinfo\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.712088 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.711951 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8qf\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-kube-api-access-2v8qf\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.812846 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.812809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-credential-socket\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.812846 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.812853 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-data\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813087 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.812875 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-socket\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813087 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.812896 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-podinfo\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813087 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.812930 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8qf\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-kube-api-access-2v8qf\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813087 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.812995 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a742c7-653c-43a5-8f0a-268d7cdb5526-istiod-ca-cert\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813087 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813025 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-envoy\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813087 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813048 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-certs\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813087 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813088 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-token\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813480 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813288 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-data\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813480 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813391 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-credential-socket\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813561 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813526 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-socket\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813625 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813559 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-certs\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.813690 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.813676 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a742c7-653c-43a5-8f0a-268d7cdb5526-istiod-ca-cert\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.815532 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.815507 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-envoy\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.815699 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.815678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-podinfo\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.826897 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.826874 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-token\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.827145 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.827128 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8qf\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-kube-api-access-2v8qf\") pod \"e2e-shared-a-824408-openshift-default-74f778f59f-p76qx\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:23.982089 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:23.981998 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:24.135376 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:24.135225 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx"] Jul 21 09:02:24.138061 ip-10-0-131-134 kubenswrapper[2553]: W0721 09:02:24.138034 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a742c7_653c_43a5_8f0a_268d7cdb5526.slice/crio-40f14421d43db6de26465aa3dec643085fdea8bff180b91b7d52ce1d04e43d20 WatchSource:0}: Error finding container 40f14421d43db6de26465aa3dec643085fdea8bff180b91b7d52ce1d04e43d20: Status 404 returned error can't find the container with id 40f14421d43db6de26465aa3dec643085fdea8bff180b91b7d52ce1d04e43d20 Jul 21 09:02:24.140112 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:24.140079 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:02:24.140220 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:24.140159 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:02:24.140220 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:24.140204 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:02:25.116874 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:25.116833 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" event={"ID":"67a742c7-653c-43a5-8f0a-268d7cdb5526","Type":"ContainerStarted","Data":"e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986"} Jul 21 09:02:25.117238 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:25.116911 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" event={"ID":"67a742c7-653c-43a5-8f0a-268d7cdb5526","Type":"ContainerStarted","Data":"40f14421d43db6de26465aa3dec643085fdea8bff180b91b7d52ce1d04e43d20"} Jul 21 09:02:25.140901 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:25.140846 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" podStartSLOduration=2.140829624 podStartE2EDuration="2.140829624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:02:25.13901294 +0000 UTC m=+2601.024683505" watchObservedRunningTime="2026-07-21 09:02:25.140829624 +0000 UTC m=+2601.026500172" Jul 21 09:02:25.982871 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:25.982833 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:25.987769 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:25.987740 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:26.121650 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:26.121612 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:26.122549 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:26.122525 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:27.404874 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.404840 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx"] Jul 21 09:02:27.408603 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.408584 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.411381 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.411358 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-824408-serving-cert\"" Jul 21 09:02:27.442842 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.442800 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx"] Jul 21 09:02:27.450537 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.450505 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.450711 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.450541 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd51d4e-e386-4842-a669-a364792e5d43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.450711 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.450591 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk47\" (UniqueName: \"kubernetes.io/projected/8fd51d4e-e386-4842-a669-a364792e5d43-kube-api-access-rqk47\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.551253 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.551220 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.551484 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.551265 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd51d4e-e386-4842-a669-a364792e5d43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.551484 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.551301 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk47\" (UniqueName: \"kubernetes.io/projected/8fd51d4e-e386-4842-a669-a364792e5d43-kube-api-access-rqk47\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.551484 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:02:27.551406 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-serving-cert: secret "maas-api-e2e-shared-a-824408-serving-cert" not found Jul 21 09:02:27.551484 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:02:27.551477 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls podName:8fd51d4e-e386-4842-a669-a364792e5d43 nodeName:}" failed. No retries permitted until 2026-07-21 09:02:28.051458549 +0000 UTC m=+2603.937129076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls") pod "maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" (UID: "8fd51d4e-e386-4842-a669-a364792e5d43") : secret "maas-api-e2e-shared-a-824408-serving-cert" not found Jul 21 09:02:27.551928 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.551909 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd51d4e-e386-4842-a669-a364792e5d43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:27.573013 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:27.572990 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk47\" (UniqueName: \"kubernetes.io/projected/8fd51d4e-e386-4842-a669-a364792e5d43-kube-api-access-rqk47\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:28.056010 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:28.055976 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:28.058608 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:28.058582 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls\") pod \"maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:28.319387 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:28.319275 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:28.472578 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:28.472520 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx"] Jul 21 09:02:28.475167 ip-10-0-131-134 kubenswrapper[2553]: W0721 09:02:28.475142 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd51d4e_e386_4842_a669_a364792e5d43.slice/crio-dc9e2e74de40fcec647cc0d2dbc9f3e00cd3b947484a301c2bc5a85315a10ba7 WatchSource:0}: Error finding container dc9e2e74de40fcec647cc0d2dbc9f3e00cd3b947484a301c2bc5a85315a10ba7: Status 404 returned error can't find the container with id dc9e2e74de40fcec647cc0d2dbc9f3e00cd3b947484a301c2bc5a85315a10ba7 Jul 21 09:02:29.138873 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:29.138770 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" event={"ID":"8fd51d4e-e386-4842-a669-a364792e5d43","Type":"ContainerStarted","Data":"9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704"} Jul 21 09:02:29.138873 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:29.138826 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" event={"ID":"8fd51d4e-e386-4842-a669-a364792e5d43","Type":"ContainerStarted","Data":"dc9e2e74de40fcec647cc0d2dbc9f3e00cd3b947484a301c2bc5a85315a10ba7"} Jul 21 09:02:29.138873 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:29.138850 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:29.162164 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:29.162094 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" podStartSLOduration=1.775518791 podStartE2EDuration="2.16207656s" totalImagesPullingTime="386.557769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:02:29.160052557 +0000 UTC m=+2605.045723104" watchObservedRunningTime="2026-07-21 09:02:29.16207656 +0000 UTC m=+2605.047747108" Jul 21 09:02:35.149391 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.149361 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:02:35.331092 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.331006 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt"] Jul 21 09:02:35.335016 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.334992 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.340548 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.340521 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-26961c-serving-cert\"" Jul 21 09:02:35.359511 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.359481 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt"] Jul 21 09:02:35.424794 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.424764 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1447c286-3fed-4b20-8082-c0dd49f9c4fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.424971 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.424825 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1447c286-3fed-4b20-8082-c0dd49f9c4fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.424971 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.424890 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2tw\" (UniqueName: \"kubernetes.io/projected/1447c286-3fed-4b20-8082-c0dd49f9c4fd-kube-api-access-qg2tw\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.525724 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.525688 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1447c286-3fed-4b20-8082-c0dd49f9c4fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.525926 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.525777 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2tw\" (UniqueName: \"kubernetes.io/projected/1447c286-3fed-4b20-8082-c0dd49f9c4fd-kube-api-access-qg2tw\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.525926 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.525827 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1447c286-3fed-4b20-8082-c0dd49f9c4fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.526433 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.526408 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1447c286-3fed-4b20-8082-c0dd49f9c4fd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.528411 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.528378 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1447c286-3fed-4b20-8082-c0dd49f9c4fd-maas-api-tls\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.539687 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.539661 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2tw\" (UniqueName: \"kubernetes.io/projected/1447c286-3fed-4b20-8082-c0dd49f9c4fd-kube-api-access-qg2tw\") pod \"maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.645646 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.645551 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:35.806712 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:35.806547 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt"] Jul 21 09:02:35.809275 ip-10-0-131-134 kubenswrapper[2553]: W0721 09:02:35.809246 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1447c286_3fed_4b20_8082_c0dd49f9c4fd.slice/crio-5d345e577c5d7f8984f8c252c04db056e3f03efa1b5db91af92c45e000ac7165 WatchSource:0}: Error finding container 5d345e577c5d7f8984f8c252c04db056e3f03efa1b5db91af92c45e000ac7165: Status 404 returned error can't find the container with id 5d345e577c5d7f8984f8c252c04db056e3f03efa1b5db91af92c45e000ac7165 Jul 21 09:02:36.174809 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:36.174772 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" event={"ID":"1447c286-3fed-4b20-8082-c0dd49f9c4fd","Type":"ContainerStarted","Data":"5d345e577c5d7f8984f8c252c04db056e3f03efa1b5db91af92c45e000ac7165"} Jul 21 09:02:37.182093 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:37.182050 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" event={"ID":"1447c286-3fed-4b20-8082-c0dd49f9c4fd","Type":"ContainerStarted","Data":"e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b"} Jul 21 09:02:37.182594 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:37.182122 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:42.163505 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:42.163459 2553 patch_prober.go:28] interesting pod/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 09:02:42.163967 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:42.163540 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 09:02:43.196283 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.196250 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:02:43.215553 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.215502 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" podStartSLOduration=7.750304271 podStartE2EDuration="8.21548498s" totalImagesPullingTime="465.180709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 09:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:02:37.214198361 +0000 UTC m=+2613.099868908" watchObservedRunningTime="2026-07-21 09:02:43.21548498 +0000 UTC m=+2619.101155527" Jul 21 09:02:43.234983 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.234935 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" event={"ID":"67a742c7-653c-43a5-8f0a-268d7cdb5526","Type":"ContainerDied","Data":"e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986"} Jul 21 09:02:43.235163 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.234931 2553 generic.go:356] "Generic (PLEG): container finished" podID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerID="e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986" exitCode=0 Jul 21 09:02:43.235534 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.235511 2553 scope.go:117] "RemoveContainer" containerID="e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986" Jul 21 09:02:43.236842 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.236808 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:02:43.236913 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.236898 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:02:43.236991 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:43.236973 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 09:02:44.241949 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:44.241905 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" event={"ID":"67a742c7-653c-43a5-8f0a-268d7cdb5526","Type":"ContainerStarted","Data":"462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4"} Jul 21 09:02:44.983130 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:44.983094 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:44.983450 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:44.983351 2553 patch_prober.go:28] interesting pod/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 09:02:44.983635 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:44.983607 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 09:02:45.982789 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:45.982749 2553 patch_prober.go:28] interesting pod/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 09:02:45.983408 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:45.983373 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 09:02:46.983391 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:46.983350 2553 patch_prober.go:28] interesting pod/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 09:02:46.984009 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:46.983971 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 09:02:48.000348 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:47.999450 2553 patch_prober.go:28] interesting pod/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 21 09:02:48.000348 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:47.999527 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 21 09:02:48.986378 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:48.986338 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:48.986719 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:48.986700 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:02:48.987326 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:02:48.987291 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:04:05.177910 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:04:05.177814 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 09:04:05.183585 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:04:05.183561 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 09:05:05.395861 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:05:05.395831 2553 scope.go:117] "RemoveContainer" containerID="b46a9b5dd5d063b8098a922a15cfc2e7bd08cbff2073529644a0e280ccae56e3" Jul 21 09:09:05.207830 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:05.207803 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 09:09:05.215723 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:05.215701 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bf86t_b20b6770-0988-4982-9540-740ef9289986/ovn-acl-logging/0.log" Jul 21 09:09:08.849080 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:08.849045 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx"] Jul 21 09:09:08.849532 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:08.849262 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" podUID="8fd51d4e-e386-4842-a669-a364792e5d43" containerName="maas-api" containerID="cri-o://9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704" gracePeriod=30 Jul 21 09:09:09.105013 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.104944 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:09:09.236376 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.236347 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls\") pod \"8fd51d4e-e386-4842-a669-a364792e5d43\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " Jul 21 09:09:09.236561 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.236436 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd51d4e-e386-4842-a669-a364792e5d43-openshift-service-ca\") pod \"8fd51d4e-e386-4842-a669-a364792e5d43\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " Jul 21 09:09:09.236561 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.236486 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqk47\" (UniqueName: \"kubernetes.io/projected/8fd51d4e-e386-4842-a669-a364792e5d43-kube-api-access-rqk47\") pod \"8fd51d4e-e386-4842-a669-a364792e5d43\" (UID: \"8fd51d4e-e386-4842-a669-a364792e5d43\") " Jul 21 09:09:09.236862 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.236832 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fd51d4e-e386-4842-a669-a364792e5d43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fd51d4e-e386-4842-a669-a364792e5d43" (UID: "8fd51d4e-e386-4842-a669-a364792e5d43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:09:09.238840 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.238805 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fd51d4e-e386-4842-a669-a364792e5d43" (UID: "8fd51d4e-e386-4842-a669-a364792e5d43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:09:09.239070 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.239054 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fd51d4e-e386-4842-a669-a364792e5d43-kube-api-access-rqk47" (OuterVolumeSpecName: "kube-api-access-rqk47") pod "8fd51d4e-e386-4842-a669-a364792e5d43" (UID: "8fd51d4e-e386-4842-a669-a364792e5d43"). InnerVolumeSpecName "kube-api-access-rqk47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:09:09.242196 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.242168 2553 generic.go:356] "Generic (PLEG): container finished" podID="8fd51d4e-e386-4842-a669-a364792e5d43" containerID="9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704" exitCode=0 Jul 21 09:09:09.242357 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.242212 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" Jul 21 09:09:09.242357 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.242250 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" event={"ID":"8fd51d4e-e386-4842-a669-a364792e5d43","Type":"ContainerDied","Data":"9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704"} Jul 21 09:09:09.242357 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.242335 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx" event={"ID":"8fd51d4e-e386-4842-a669-a364792e5d43","Type":"ContainerDied","Data":"dc9e2e74de40fcec647cc0d2dbc9f3e00cd3b947484a301c2bc5a85315a10ba7"} Jul 21 09:09:09.242544 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.242367 2553 scope.go:117] "RemoveContainer" containerID="9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704" Jul 21 09:09:09.257928 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.257905 2553 scope.go:117] "RemoveContainer" containerID="9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704" Jul 21 09:09:09.258255 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:09:09.258228 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704\": container with ID starting with 9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704 not found: ID does not exist" containerID="9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704" Jul 21 09:09:09.258367 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.258269 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704"} err="failed to get container status \"9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704\": rpc error: code = NotFound desc = could not find container \"9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704\": container with ID starting with 9b0337c3b9a780e676d74be352eb4ffd34bd80e2e936fcee169f7849c9e01704 not found: ID does not exist" Jul 21 09:09:09.268738 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.268708 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx"] Jul 21 09:09:09.270586 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.270565 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-824408-5f48bc95db-z5bgx"] Jul 21 09:09:09.338017 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.337980 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqk47\" (UniqueName: \"kubernetes.io/projected/8fd51d4e-e386-4842-a669-a364792e5d43-kube-api-access-rqk47\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:09.338017 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.338011 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fd51d4e-e386-4842-a669-a364792e5d43-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:09.338017 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:09.338023 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fd51d4e-e386-4842-a669-a364792e5d43-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:10.674627 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:10.674592 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fd51d4e-e386-4842-a669-a364792e5d43" path="/var/lib/kubelet/pods/8fd51d4e-e386-4842-a669-a364792e5d43/volumes" Jul 21 09:09:20.544146 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:20.544114 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx"] Jul 21 09:09:20.544617 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:20.544340 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" containerID="cri-o://462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4" gracePeriod=30 Jul 21 09:09:25.801837 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.801807 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:09:25.893198 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893101 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-credential-socket\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893198 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893153 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-socket\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893198 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893183 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a742c7-653c-43a5-8f0a-268d7cdb5526-istiod-ca-cert\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893529 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893205 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-envoy\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893529 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893222 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-data\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893529 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893258 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v8qf\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-kube-api-access-2v8qf\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893529 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893358 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-certs\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893529 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893381 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-token\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893529 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893421 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-podinfo\") pod \"67a742c7-653c-43a5-8f0a-268d7cdb5526\" (UID: \"67a742c7-653c-43a5-8f0a-268d7cdb5526\") " Jul 21 09:09:25.893529 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893459 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:09:25.893882 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893555 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:09:25.893882 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893678 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a742c7-653c-43a5-8f0a-268d7cdb5526-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:09:25.893882 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893739 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-credential-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.893882 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893767 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-socket\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.894094 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.893894 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:09:25.894330 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.894272 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-data" (OuterVolumeSpecName: "istio-data") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:09:25.895862 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.895834 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-token" (OuterVolumeSpecName: "istio-token") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:09:25.895862 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.895850 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 09:09:25.896061 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.896038 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-kube-api-access-2v8qf" (OuterVolumeSpecName: "kube-api-access-2v8qf") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "kube-api-access-2v8qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:09:25.896114 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.896100 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67a742c7-653c-43a5-8f0a-268d7cdb5526" (UID: "67a742c7-653c-43a5-8f0a-268d7cdb5526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 09:09:25.994651 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.994597 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67a742c7-653c-43a5-8f0a-268d7cdb5526-istiod-ca-cert\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.994651 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.994645 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-envoy\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.994651 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.994657 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-data\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.994912 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.994669 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v8qf\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-kube-api-access-2v8qf\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.994912 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.994684 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67a742c7-653c-43a5-8f0a-268d7cdb5526-workload-certs\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.994912 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.994695 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-token\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:25.994912 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:25.994705 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67a742c7-653c-43a5-8f0a-268d7cdb5526-istio-podinfo\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:26.332134 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.332099 2553 generic.go:356] "Generic (PLEG): container finished" podID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerID="462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4" exitCode=0 Jul 21 09:09:26.332344 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.332146 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" Jul 21 09:09:26.332344 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.332181 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" event={"ID":"67a742c7-653c-43a5-8f0a-268d7cdb5526","Type":"ContainerDied","Data":"462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4"} Jul 21 09:09:26.332344 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.332236 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx" event={"ID":"67a742c7-653c-43a5-8f0a-268d7cdb5526","Type":"ContainerDied","Data":"40f14421d43db6de26465aa3dec643085fdea8bff180b91b7d52ce1d04e43d20"} Jul 21 09:09:26.332344 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.332267 2553 scope.go:117] "RemoveContainer" containerID="462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4" Jul 21 09:09:26.341774 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.341757 2553 scope.go:117] "RemoveContainer" containerID="e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986" Jul 21 09:09:26.351533 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.351514 2553 scope.go:117] "RemoveContainer" containerID="462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4" Jul 21 09:09:26.351803 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:09:26.351780 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4\": container with ID starting with 462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4 not found: ID does not exist" containerID="462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4" Jul 21 09:09:26.351851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.351813 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4"} err="failed to get container status \"462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4\": rpc error: code = NotFound desc = could not find container \"462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4\": container with ID starting with 462256e95faae89e4cba05288087115778adda9f31099d49fcb82c66012042e4 not found: ID does not exist" Jul 21 09:09:26.351851 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.351836 2553 scope.go:117] "RemoveContainer" containerID="e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986" Jul 21 09:09:26.352076 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:09:26.352050 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986\": container with ID starting with e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986 not found: ID does not exist" containerID="e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986" Jul 21 09:09:26.352117 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.352083 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986"} err="failed to get container status \"e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986\": rpc error: code = NotFound desc = could not find container \"e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986\": container with ID starting with e6e6eb9f66a083cbdc0a4a0e2e7d5fdf4bb72ca4fd38d6f3206e5f25e55ac986 not found: ID does not exist" Jul 21 09:09:26.355472 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.355452 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx"] Jul 21 09:09:26.360302 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.360283 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-824408-openshift-default-74f778f59f-p76qx"] Jul 21 09:09:26.674487 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:26.674454 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" path="/var/lib/kubelet/pods/67a742c7-653c-43a5-8f0a-268d7cdb5526/volumes" Jul 21 09:09:31.442270 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.442236 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt"] Jul 21 09:09:31.442781 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.442576 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" podUID="1447c286-3fed-4b20-8082-c0dd49f9c4fd" containerName="maas-api" containerID="cri-o://e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b" gracePeriod=30 Jul 21 09:09:31.703464 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.703401 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:09:31.848435 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.848397 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1447c286-3fed-4b20-8082-c0dd49f9c4fd-maas-api-tls\") pod \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " Jul 21 09:09:31.848435 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.848442 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg2tw\" (UniqueName: \"kubernetes.io/projected/1447c286-3fed-4b20-8082-c0dd49f9c4fd-kube-api-access-qg2tw\") pod \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " Jul 21 09:09:31.848670 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.848579 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1447c286-3fed-4b20-8082-c0dd49f9c4fd-openshift-service-ca\") pod \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\" (UID: \"1447c286-3fed-4b20-8082-c0dd49f9c4fd\") " Jul 21 09:09:31.848947 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.848923 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1447c286-3fed-4b20-8082-c0dd49f9c4fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1447c286-3fed-4b20-8082-c0dd49f9c4fd" (UID: "1447c286-3fed-4b20-8082-c0dd49f9c4fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 09:09:31.850643 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.850618 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1447c286-3fed-4b20-8082-c0dd49f9c4fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1447c286-3fed-4b20-8082-c0dd49f9c4fd" (UID: "1447c286-3fed-4b20-8082-c0dd49f9c4fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 09:09:31.850643 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.850625 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1447c286-3fed-4b20-8082-c0dd49f9c4fd-kube-api-access-qg2tw" (OuterVolumeSpecName: "kube-api-access-qg2tw") pod "1447c286-3fed-4b20-8082-c0dd49f9c4fd" (UID: "1447c286-3fed-4b20-8082-c0dd49f9c4fd"). InnerVolumeSpecName "kube-api-access-qg2tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 09:09:31.949552 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.949510 2553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1447c286-3fed-4b20-8082-c0dd49f9c4fd-openshift-service-ca\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:31.949552 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.949544 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1447c286-3fed-4b20-8082-c0dd49f9c4fd-maas-api-tls\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:31.949552 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:31.949554 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg2tw\" (UniqueName: \"kubernetes.io/projected/1447c286-3fed-4b20-8082-c0dd49f9c4fd-kube-api-access-qg2tw\") on node \"ip-10-0-131-134.ec2.internal\" DevicePath \"\"" Jul 21 09:09:32.364522 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.364487 2553 generic.go:356] "Generic (PLEG): container finished" podID="1447c286-3fed-4b20-8082-c0dd49f9c4fd" containerID="e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b" exitCode=0 Jul 21 09:09:32.364711 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.364533 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" Jul 21 09:09:32.364711 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.364557 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" event={"ID":"1447c286-3fed-4b20-8082-c0dd49f9c4fd","Type":"ContainerDied","Data":"e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b"} Jul 21 09:09:32.364711 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.364600 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt" event={"ID":"1447c286-3fed-4b20-8082-c0dd49f9c4fd","Type":"ContainerDied","Data":"5d345e577c5d7f8984f8c252c04db056e3f03efa1b5db91af92c45e000ac7165"} Jul 21 09:09:32.364711 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.364621 2553 scope.go:117] "RemoveContainer" containerID="e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b" Jul 21 09:09:32.374518 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.374499 2553 scope.go:117] "RemoveContainer" containerID="e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b" Jul 21 09:09:32.374779 ip-10-0-131-134 kubenswrapper[2553]: E0721 09:09:32.374760 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b\": container with ID starting with e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b not found: ID does not exist" containerID="e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b" Jul 21 09:09:32.374828 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.374789 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b"} err="failed to get container status \"e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b\": rpc error: code = NotFound desc = could not find container \"e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b\": container with ID starting with e665b0df5895a1efd4fd765e423f2dbb76f094230f935dda1f858f27a295bf0b not found: ID does not exist" Jul 21 09:09:32.388686 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.388654 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt"] Jul 21 09:09:32.392433 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.392409 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-26961c-7c599d8596-fkxzt"] Jul 21 09:09:32.676895 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:32.676852 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1447c286-3fed-4b20-8082-c0dd49f9c4fd" path="/var/lib/kubelet/pods/1447c286-3fed-4b20-8082-c0dd49f9c4fd/volumes" Jul 21 09:09:43.820931 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:43.820890 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6549bdf6bc-l2pfx_4d92c825-609a-40dc-8968-f77697b8fda8/authorino/0.log" Jul 21 09:09:49.554302 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:49.554258 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hmqbx_33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a/manager/0.log" Jul 21 09:09:49.672050 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:49.672018 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jk7dw_932abbba-9568-4652-b6c3-bc0d9973b14a/manager/0.log" Jul 21 09:09:49.798086 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:49.798036 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68d799f9c8-4glzb_7f76cd9b-efd8-4e60-8e76-f87f779db044/manager/0.log" Jul 21 09:09:49.905091 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:49.904985 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zbfhp_4e8dbd59-fb17-42c1-b8ab-b82b75142d11/server/0.log" Jul 21 09:09:50.028232 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:50.028204 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h7s4c_4ed5bb95-54c6-4878-97c9-aa8ea6cc946e/manager/0.log" Jul 21 09:09:50.288453 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:50.288425 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-r45tb_8ca8fbc0-8b40-461e-997d-8f03052c473e/copy-manifests/0.log" Jul 21 09:09:50.295033 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:50.295011 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-r45tb_8ca8fbc0-8b40-461e-997d-8f03052c473e/manager/0.log" Jul 21 09:09:50.801464 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:50.801432 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86f46b5fc6-swpsr_49cf2062-682e-4e5c-bad8-763f624e557d/maas-api/0.log" Jul 21 09:09:51.168255 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:51.168229 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zlt99_e2c87943-9a89-4b9e-a8d4-f9b4f1909890/postgres/0.log" Jul 21 09:09:52.508474 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:52.508440 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6549bdf6bc-l2pfx_4d92c825-609a-40dc-8968-f77697b8fda8/authorino/0.log" Jul 21 09:09:52.624518 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:52.624488 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gjwt2_b34b31f5-3c43-4a36-9dbc-eb34c976c05a/manager/0.log" Jul 21 09:09:52.982036 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:52.981997 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dtc5h_72a37a62-b58f-4410-b29e-49b88d4dd938/registry-server/0.log" Jul 21 09:09:53.326235 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:53.326161 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-g698r_5aabb72b-458c-4113-86ce-d99307b69515/manager/0.log" Jul 21 09:09:53.682943 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:53.682912 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6784n74f_8e0dd0b4-f3b4-4460-9944-9c5e01b188e1/istio-proxy/0.log" Jul 21 09:09:54.022947 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:54.022865 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-cwctz_e23b9e72-2997-429f-8c4c-63789ddbf0c5/kube-auth-proxy/0.log" Jul 21 09:09:54.139478 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:54.139449 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-dk6s5_9eb90821-604b-40ab-8efb-f82ca783417a/istio-proxy/1.log" Jul 21 09:09:54.256290 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:54.256261 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bmsts_c45f1607-5111-4a02-953e-64a20cda6cd3/payload-pre-processing/0.log" Jul 21 09:09:54.367890 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:54.367808 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-ltznq_3e51bae1-88cf-461b-b19e-4395574fba30/payload-processing/0.log" Jul 21 09:09:54.844536 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:54.844502 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-85jcl_bdf15937-3c9b-4526-b821-12fd309b5f36/main/0.log" Jul 21 09:09:54.965439 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:09:54.965410 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bv6dn_e95a8885-61f4-48ff-8b46-aff8e4de979f/main/0.log" Jul 21 09:10:04.373559 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:04.373528 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2ffz9_d8a24c11-99e9-4221-b50b-851f24298d13/global-pull-secret-syncer/0.log" Jul 21 09:10:04.552662 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:04.552635 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pfwh9_d35d5d7a-78b8-4f4c-947d-d738b21f2034/konnectivity-agent/0.log" Jul 21 09:10:04.610814 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:04.610785 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-134.ec2.internal_420479272364a70b28d1192701ae6b13/haproxy/0.log" Jul 21 09:10:10.042669 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:10.042633 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6549bdf6bc-l2pfx_4d92c825-609a-40dc-8968-f77697b8fda8/authorino/0.log" Jul 21 09:10:10.078883 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:10.078844 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gjwt2_b34b31f5-3c43-4a36-9dbc-eb34c976c05a/manager/0.log" Jul 21 09:10:10.218873 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:10.218843 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dtc5h_72a37a62-b58f-4410-b29e-49b88d4dd938/registry-server/0.log" Jul 21 09:10:10.488711 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:10.488677 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-g698r_5aabb72b-458c-4113-86ce-d99307b69515/manager/0.log" Jul 21 09:10:12.659633 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:12.659607 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-ttvth_0ece16ad-237d-4d25-bf60-12faf52b229e/monitoring-plugin/0.log" Jul 21 09:10:12.721948 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:12.721919 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkdwl_3d769311-4e69-43c3-95a4-1be71d1bbe14/node-exporter/0.log" Jul 21 09:10:12.754729 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:12.754701 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkdwl_3d769311-4e69-43c3-95a4-1be71d1bbe14/kube-rbac-proxy/0.log" Jul 21 09:10:12.803435 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:12.803406 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkdwl_3d769311-4e69-43c3-95a4-1be71d1bbe14/init-textfile/0.log" Jul 21 09:10:13.383372 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.383342 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9891a4e3-56c2-450c-b47d-611674391b2f/prometheus/0.log" Jul 21 09:10:13.418371 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.418340 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9891a4e3-56c2-450c-b47d-611674391b2f/config-reloader/0.log" Jul 21 09:10:13.463536 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.463504 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9891a4e3-56c2-450c-b47d-611674391b2f/thanos-sidecar/0.log" Jul 21 09:10:13.507128 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.507105 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9891a4e3-56c2-450c-b47d-611674391b2f/kube-rbac-proxy-web/0.log" Jul 21 09:10:13.548559 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.548526 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9891a4e3-56c2-450c-b47d-611674391b2f/kube-rbac-proxy/0.log" Jul 21 09:10:13.603518 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.603489 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9891a4e3-56c2-450c-b47d-611674391b2f/kube-rbac-proxy-thanos/0.log" Jul 21 09:10:13.638754 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.638677 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9891a4e3-56c2-450c-b47d-611674391b2f/init-config-reloader/0.log" Jul 21 09:10:13.926097 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.926068 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6688d859f-s89c8_a85ff1a4-22d9-44f0-be89-bc146b606b19/thanos-query/0.log" Jul 21 09:10:13.966812 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:13.966783 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6688d859f-s89c8_a85ff1a4-22d9-44f0-be89-bc146b606b19/kube-rbac-proxy-web/0.log" Jul 21 09:10:14.000558 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.000532 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6688d859f-s89c8_a85ff1a4-22d9-44f0-be89-bc146b606b19/kube-rbac-proxy/0.log" Jul 21 09:10:14.038596 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.038570 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6688d859f-s89c8_a85ff1a4-22d9-44f0-be89-bc146b606b19/prom-label-proxy/0.log" Jul 21 09:10:14.075211 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.075183 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6688d859f-s89c8_a85ff1a4-22d9-44f0-be89-bc146b606b19/kube-rbac-proxy-rules/0.log" Jul 21 09:10:14.110220 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.110196 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6688d859f-s89c8_a85ff1a4-22d9-44f0-be89-bc146b606b19/kube-rbac-proxy-metrics/0.log" Jul 21 09:10:14.465406 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465364 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8"] Jul 21 09:10:14.465846 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465829 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fd51d4e-e386-4842-a669-a364792e5d43" containerName="maas-api" Jul 21 09:10:14.465944 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465850 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fd51d4e-e386-4842-a669-a364792e5d43" containerName="maas-api" Jul 21 09:10:14.465944 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465873 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1447c286-3fed-4b20-8082-c0dd49f9c4fd" containerName="maas-api" Jul 21 09:10:14.465944 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465881 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1447c286-3fed-4b20-8082-c0dd49f9c4fd" containerName="maas-api" Jul 21 09:10:14.465944 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465892 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" Jul 21 09:10:14.465944 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465900 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" Jul 21 09:10:14.465944 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465925 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" Jul 21 09:10:14.465944 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.465934 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" Jul 21 09:10:14.466301 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.466035 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" Jul 21 09:10:14.466301 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.466046 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a742c7-653c-43a5-8f0a-268d7cdb5526" containerName="istio-proxy" Jul 21 09:10:14.466301 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.466058 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fd51d4e-e386-4842-a669-a364792e5d43" containerName="maas-api" Jul 21 09:10:14.466301 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.466067 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1447c286-3fed-4b20-8082-c0dd49f9c4fd" containerName="maas-api" Jul 21 09:10:14.469521 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.469498 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.472302 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.472279 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xts55\"/\"kube-root-ca.crt\"" Jul 21 09:10:14.473206 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.473183 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xts55\"/\"openshift-service-ca.crt\"" Jul 21 09:10:14.473295 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.473191 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xts55\"/\"default-dockercfg-zhkmm\"" Jul 21 09:10:14.488778 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.488753 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8"] Jul 21 09:10:14.549786 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.549744 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-lib-modules\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.549966 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.549795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj4dz\" (UniqueName: \"kubernetes.io/projected/911ac81b-a9cf-4dae-b327-49afe142f3ab-kube-api-access-rj4dz\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.549966 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.549833 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-podres\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.549966 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.549858 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-proc\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.549966 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.549906 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-sys\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.650692 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650659 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-lib-modules\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.650902 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650701 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj4dz\" (UniqueName: \"kubernetes.io/projected/911ac81b-a9cf-4dae-b327-49afe142f3ab-kube-api-access-rj4dz\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.650902 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650737 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-podres\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.650902 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-proc\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.650902 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650792 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-sys\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.650902 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650877 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-sys\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.650902 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650876 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-lib-modules\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.651129 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650901 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-podres\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.651129 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.650900 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/911ac81b-a9cf-4dae-b327-49afe142f3ab-proc\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.660170 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.660141 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj4dz\" (UniqueName: \"kubernetes.io/projected/911ac81b-a9cf-4dae-b327-49afe142f3ab-kube-api-access-rj4dz\") pod \"perf-node-gather-daemonset-4mgx8\" (UID: \"911ac81b-a9cf-4dae-b327-49afe142f3ab\") " pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.779589 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.779483 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:14.920232 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:14.920205 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8"] Jul 21 09:10:14.922510 ip-10-0-131-134 kubenswrapper[2553]: W0721 09:10:14.922480 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod911ac81b_a9cf_4dae_b327_49afe142f3ab.slice/crio-71f089713cd0dc44bbdbf36e6b06c5a794ff7a058d11127fb9a0a5e72cbd476b WatchSource:0}: Error finding container 71f089713cd0dc44bbdbf36e6b06c5a794ff7a058d11127fb9a0a5e72cbd476b: Status 404 returned error can't find the container with id 71f089713cd0dc44bbdbf36e6b06c5a794ff7a058d11127fb9a0a5e72cbd476b Jul 21 09:10:15.577264 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:15.577221 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" event={"ID":"911ac81b-a9cf-4dae-b327-49afe142f3ab","Type":"ContainerStarted","Data":"f93d922271795d310b0978e4bfc9cb9cad4821ee405d4be310affecc14eb1b16"} Jul 21 09:10:15.577718 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:15.577285 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" event={"ID":"911ac81b-a9cf-4dae-b327-49afe142f3ab","Type":"ContainerStarted","Data":"71f089713cd0dc44bbdbf36e6b06c5a794ff7a058d11127fb9a0a5e72cbd476b"} Jul 21 09:10:15.577718 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:15.577429 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:15.593674 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:15.593626 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" podStartSLOduration=1.5936091449999998 podStartE2EDuration="1.593609145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 09:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 09:10:15.593009465 +0000 UTC m=+3071.478680014" watchObservedRunningTime="2026-07-21 09:10:15.593609145 +0000 UTC m=+3071.479279693" Jul 21 09:10:18.176523 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:18.176498 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6dw7h_b3de9de6-8a89-44ef-8c4c-291377d55d49/dns/0.log" Jul 21 09:10:18.202061 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:18.202035 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6dw7h_b3de9de6-8a89-44ef-8c4c-291377d55d49/kube-rbac-proxy/0.log" Jul 21 09:10:18.411745 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:18.411720 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ksjr8_d775c285-f2a3-42b5-8867-88a0faefff5f/dns-node-resolver/0.log" Jul 21 09:10:19.030298 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:19.030265 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bxvjt_f2090af5-b4b4-457b-8a32-8e361407a9ea/node-ca/0.log" Jul 21 09:10:20.112846 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:20.112813 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6784n74f_8e0dd0b4-f3b4-4460-9944-9c5e01b188e1/istio-proxy/0.log" Jul 21 09:10:20.292870 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:20.292843 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-cwctz_e23b9e72-2997-429f-8c4c-63789ddbf0c5/kube-auth-proxy/0.log" Jul 21 09:10:20.327192 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:20.327165 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-dk6s5_9eb90821-604b-40ab-8efb-f82ca783417a/istio-proxy/0.log" Jul 21 09:10:20.352004 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:20.351972 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-dk6s5_9eb90821-604b-40ab-8efb-f82ca783417a/istio-proxy/1.log" Jul 21 09:10:20.381094 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:20.381024 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bmsts_c45f1607-5111-4a02-953e-64a20cda6cd3/payload-pre-processing/0.log" Jul 21 09:10:20.413207 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:20.413178 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-ltznq_3e51bae1-88cf-461b-b19e-4395574fba30/payload-processing/0.log" Jul 21 09:10:21.226479 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:21.226448 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9g7hf_14798834-275f-4627-a77c-d93602ef8a42/serve-healthcheck-canary/0.log" Jul 21 09:10:21.593234 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:21.593153 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xts55/perf-node-gather-daemonset-4mgx8" Jul 21 09:10:21.976088 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:21.976054 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tg755_55a3d257-5301-44c0-8222-9acecc1fdf19/kube-rbac-proxy/0.log" Jul 21 09:10:22.000520 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:22.000483 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tg755_55a3d257-5301-44c0-8222-9acecc1fdf19/exporter/0.log" Jul 21 09:10:22.036579 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:22.036553 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tg755_55a3d257-5301-44c0-8222-9acecc1fdf19/extractor/0.log" Jul 21 09:10:24.689671 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:24.689644 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6784n74f_8e0dd0b4-f3b4-4460-9944-9c5e01b188e1/istio-proxy/0.log" Jul 21 09:10:24.874844 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:24.874818 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-cwctz_e23b9e72-2997-429f-8c4c-63789ddbf0c5/kube-auth-proxy/0.log" Jul 21 09:10:24.900363 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:24.900333 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-dk6s5_9eb90821-604b-40ab-8efb-f82ca783417a/istio-proxy/0.log" Jul 21 09:10:24.934978 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:24.934941 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-dk6s5_9eb90821-604b-40ab-8efb-f82ca783417a/istio-proxy/1.log" Jul 21 09:10:24.951373 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:24.951342 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hmqbx_33af3ff0-8b0e-48c1-8468-fbfcde5e7b6a/manager/0.log" Jul 21 09:10:24.958046 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:24.958024 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bmsts_c45f1607-5111-4a02-953e-64a20cda6cd3/payload-pre-processing/0.log" Jul 21 09:10:24.984483 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:24.984453 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-ltznq_3e51bae1-88cf-461b-b19e-4395574fba30/payload-processing/0.log" Jul 21 09:10:25.138932 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:25.138844 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jk7dw_932abbba-9568-4652-b6c3-bc0d9973b14a/manager/0.log" Jul 21 09:10:25.243195 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:25.243164 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68d799f9c8-4glzb_7f76cd9b-efd8-4e60-8e76-f87f779db044/manager/0.log" Jul 21 09:10:25.290487 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:25.290462 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zbfhp_4e8dbd59-fb17-42c1-b8ab-b82b75142d11/server/0.log" Jul 21 09:10:25.382774 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:25.382744 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h7s4c_4ed5bb95-54c6-4878-97c9-aa8ea6cc946e/manager/0.log" Jul 21 09:10:25.567194 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:25.567167 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-r45tb_8ca8fbc0-8b40-461e-997d-8f03052c473e/manager/0.log" Jul 21 09:10:25.592145 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:25.592120 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-787895fdf5-r45tb_8ca8fbc0-8b40-461e-997d-8f03052c473e/copy-manifests/0.log" Jul 21 09:10:26.264767 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:26.264741 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6784n74f_8e0dd0b4-f3b4-4460-9944-9c5e01b188e1/istio-proxy/0.log" Jul 21 09:10:26.484341 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:26.484250 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d57bc4cd9-cwctz_e23b9e72-2997-429f-8c4c-63789ddbf0c5/kube-auth-proxy/0.log" Jul 21 09:10:26.511778 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:26.511753 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-dk6s5_9eb90821-604b-40ab-8efb-f82ca783417a/istio-proxy/0.log" Jul 21 09:10:26.537838 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:26.537815 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-dk6s5_9eb90821-604b-40ab-8efb-f82ca783417a/istio-proxy/1.log" Jul 21 09:10:26.563676 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:26.563650 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bmsts_c45f1607-5111-4a02-953e-64a20cda6cd3/payload-pre-processing/0.log" Jul 21 09:10:26.595849 ip-10-0-131-134 kubenswrapper[2553]: I0721 09:10:26.595816 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-ltznq_3e51bae1-88cf-461b-b19e-4395574fba30/payload-processing/0.log"