Jul 28 19:57:25.160096 ip-10-0-136-255 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 19:57:25.160108 ip-10-0-136-255 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 19:57:25.160118 ip-10-0-136-255 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 19:57:25.160507 ip-10-0-136-255 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 19:57:35.281479 ip-10-0-136-255 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 19:57:35.281499 ip-10-0-136-255 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b706d2f9e660427a999d8513a02d62c7 -- Jul 28 19:59:31.263092 ip-10-0-136-255 systemd[1]: Starting Kubernetes Kubelet... Jul 28 19:59:31.671111 ip-10-0-136-255 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 19:59:31.671111 ip-10-0-136-255 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 19:59:31.671111 ip-10-0-136-255 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 19:59:31.671111 ip-10-0-136-255 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 19:59:31.671111 ip-10-0-136-255 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 19:59:31.672093 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.671973 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 19:59:31.675543 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675526 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:59:31.675543 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675543 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675548 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675552 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675557 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675561 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675564 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675567 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675571 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675574 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675577 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675581 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675585 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675587 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675590 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675593 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675596 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675598 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675601 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:59:31.675611 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675604 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675606 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675609 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675612 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675614 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675621 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675624 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675627 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675629 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675632 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675634 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675637 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675640 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675642 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675645 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675647 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675650 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675652 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675655 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675657 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:59:31.676101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675660 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675663 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675666 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675668 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675671 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675673 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675676 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675679 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675681 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675684 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675686 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675689 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675691 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675694 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675697 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675700 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675702 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675706 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675710 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:59:31.676579 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675713 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675716 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675719 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675721 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675725 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675728 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675730 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675733 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675735 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675738 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675740 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675743 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675745 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675748 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675750 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675752 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675755 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675758 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675761 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675763 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:59:31.677101 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675766 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675769 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675771 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675774 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675776 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675779 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675781 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675785 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675800 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675803 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675806 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675809 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675812 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675814 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675817 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675819 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675822 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675824 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675827 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675830 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:59:31.677582 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.675832 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676570 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676577 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676580 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676583 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676587 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676590 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676593 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676596 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676599 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676601 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676604 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676607 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676610 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676612 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676615 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676618 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676620 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676625 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676630 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:59:31.678069 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676633 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676636 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676638 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676642 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676646 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676649 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676652 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676656 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676659 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676662 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676664 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676667 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676669 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676672 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676675 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676677 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676680 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676683 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676686 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:59:31.678559 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676689 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676692 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676694 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676697 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676700 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676702 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676705 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676708 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676712 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676715 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676718 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676720 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676723 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676727 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676730 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676734 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676736 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676739 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676741 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676744 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:59:31.679029 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676746 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676749 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676752 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676754 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676757 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676759 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676762 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676764 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676767 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676770 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676773 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676775 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676778 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676781 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676784 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676787 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676805 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676808 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676811 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676813 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:59:31.679558 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676816 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676819 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676821 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676825 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676828 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676831 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676833 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676836 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676839 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676841 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676844 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676846 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676849 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676851 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676854 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676857 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676860 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676862 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676865 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676867 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:59:31.680080 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.676870 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677256 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677260 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677263 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677266 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677269 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677272 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677275 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677278 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677280 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677283 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677286 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677288 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677291 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677293 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677296 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677299 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677302 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677304 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677307 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:59:31.680571 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677309 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677312 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677315 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677317 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677320 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677322 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677325 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677327 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677330 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677332 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677335 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677339 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677342 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677344 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677347 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677349 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677352 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677354 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677358 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677360 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:59:31.681066 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677363 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677366 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677368 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677371 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677375 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677379 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677382 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677385 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677388 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677391 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677393 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677396 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677398 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677401 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677403 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677406 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677409 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677411 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677414 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:59:31.681541 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677416 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677419 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677421 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677424 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677429 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677431 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677434 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677436 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677439 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677442 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677445 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677447 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677450 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677453 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677456 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677459 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677462 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677464 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677467 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677469 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:59:31.682022 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677472 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677475 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677477 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677481 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677484 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677486 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677489 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677492 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677494 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677497 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677499 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677502 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677505 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677508 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677510 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677513 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677518 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677522 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677525 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:59:31.682509 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677528 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.677530 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677601 2581 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677608 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677615 2581 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677619 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677624 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677627 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677632 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677636 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677640 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677642 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677646 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677650 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677653 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677656 2581 flags.go:64] FLAG: --cgroup-root="" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677659 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677662 2581 flags.go:64] FLAG: --client-ca-file="" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677665 2581 flags.go:64] FLAG: --cloud-provider="external" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677668 2581 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677673 2581 flags.go:64] FLAG: --cluster-domain="" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677676 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677679 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677682 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 19:59:31.682996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677685 2581 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677689 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677692 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677695 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677699 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677704 2581 flags.go:64] FLAG: --contention-profiling="false" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677707 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677710 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677713 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677716 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677720 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677723 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677726 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677729 2581 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677732 2581 flags.go:64] FLAG: --enable-server="true" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677736 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677740 2581 flags.go:64] FLAG: --event-burst="100" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677744 2581 flags.go:64] FLAG: --event-qps="50" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677747 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677750 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677753 2581 flags.go:64] FLAG: --eviction-hard="" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677757 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677760 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677764 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677767 2581 flags.go:64] FLAG: --eviction-soft="" Jul 28 19:59:31.683574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677770 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677773 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677776 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677779 2581 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677782 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677785 2581 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677787 2581 flags.go:64] FLAG: --feature-gates="" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677805 2581 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677808 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677811 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677815 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677818 2581 flags.go:64] FLAG: --healthz-port="10248" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677827 2581 flags.go:64] FLAG: --help="false" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677830 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-136-255.ec2.internal" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677833 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677837 2581 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677840 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677843 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677847 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677850 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677853 2581 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677856 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677859 2581 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 19:59:31.684197 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677862 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677865 2581 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677868 2581 flags.go:64] FLAG: --kube-reserved="" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677871 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677874 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677877 2581 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677880 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677883 2581 flags.go:64] FLAG: --lock-file="" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677886 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677889 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677892 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677897 2581 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677900 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677902 2581 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677905 2581 flags.go:64] FLAG: --logging-format="text" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677908 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677911 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677914 2581 flags.go:64] FLAG: --manifest-url="" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677917 2581 flags.go:64] FLAG: --manifest-url-header="" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677921 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677924 2581 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677931 2581 flags.go:64] FLAG: --max-pods="110" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677935 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677938 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677940 2581 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 19:59:31.684777 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677944 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677947 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677950 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677953 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677960 2581 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677963 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677966 2581 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677969 2581 flags.go:64] FLAG: --pod-cidr="" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677972 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677977 2581 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677980 2581 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677983 2581 flags.go:64] FLAG: --pods-per-core="0" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677986 2581 flags.go:64] FLAG: --port="10250" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677989 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677992 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08def7038507ab622" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677996 2581 flags.go:64] FLAG: --qos-reserved="" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.677999 2581 flags.go:64] FLAG: --read-only-port="10255" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678002 2581 flags.go:64] FLAG: --register-node="true" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678004 2581 flags.go:64] FLAG: --register-with-taints="" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678013 2581 flags.go:64] FLAG: --registry-burst="10" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678016 2581 flags.go:64] FLAG: --registry-qps="5" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678019 2581 flags.go:64] FLAG: --reserved-cpus="" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678021 2581 flags.go:64] FLAG: --reserved-memory="" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678025 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678028 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 19:59:31.685594 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678031 2581 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678034 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678037 2581 flags.go:64] FLAG: --runonce="false" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678045 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678050 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678053 2581 flags.go:64] FLAG: --seccomp-default="false" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678056 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678059 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678062 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678065 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678068 2581 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678072 2581 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678075 2581 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678078 2581 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678081 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678084 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678088 2581 flags.go:64] FLAG: --system-cgroups="" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678091 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678096 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678099 2581 flags.go:64] FLAG: --tls-cert-file="" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678102 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678106 2581 flags.go:64] FLAG: --tls-min-version="" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678108 2581 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678111 2581 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678114 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 19:59:31.686217 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678117 2581 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678120 2581 flags.go:64] FLAG: --v="2" Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678124 2581 flags.go:64] FLAG: --version="false" Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678128 2581 flags.go:64] FLAG: --vmodule="" Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678132 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.678135 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678247 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678251 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678254 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678257 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678261 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678265 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678268 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678272 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678277 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678280 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678283 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678286 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678289 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678292 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:59:31.686882 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678295 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678297 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678300 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678303 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678306 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678309 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678311 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678314 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678317 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678320 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678322 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678325 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678327 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678330 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678332 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678335 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678337 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678340 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678342 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678345 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:59:31.687367 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678347 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678350 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678354 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678358 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678361 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678363 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678366 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678369 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678371 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678374 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678376 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678379 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678382 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678384 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678387 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678389 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678392 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678395 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678397 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678400 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:59:31.687881 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678402 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678405 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678408 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678410 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678413 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678415 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678418 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678420 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678423 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678426 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678428 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678431 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678433 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678436 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678440 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678444 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678446 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678449 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678451 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678454 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:59:31.688377 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678456 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678459 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678461 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678464 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678467 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678469 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678473 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678476 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678479 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678482 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678486 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678488 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678491 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678493 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678496 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678499 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678502 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678505 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678508 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:59:31.688940 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678510 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678513 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678516 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678518 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678521 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.678523 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.679398 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.686845 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.686859 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.687616 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.687626 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687706 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687711 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687715 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687718 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:59:31.689408 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687721 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687723 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687726 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687729 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687731 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687734 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687737 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687740 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687743 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687746 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687749 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687751 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687754 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687756 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687759 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687761 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687764 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687767 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687769 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:59:31.689779 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687772 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687774 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687777 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687779 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687782 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687785 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687788 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687805 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687808 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687811 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687815 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687819 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687822 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687825 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687829 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687831 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687835 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687838 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687841 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687843 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:59:31.690269 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687846 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687848 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687852 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687854 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687857 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687859 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687862 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687864 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687867 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687871 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687875 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687878 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687880 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687883 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687886 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687889 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687892 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687895 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687897 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687900 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:59:31.690774 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687903 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687905 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687908 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687910 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687913 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687915 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687918 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687920 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687923 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687925 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687928 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687930 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687933 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687935 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687944 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687950 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687953 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687956 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687959 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687962 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:59:31.691304 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687964 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687967 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687969 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687972 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687974 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687977 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687980 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687982 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687985 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687988 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687990 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687993 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687996 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.687998 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688001 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:59:31.691784 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688004 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.688008 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688120 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688124 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688127 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688130 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688133 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688136 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688138 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688141 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688143 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688146 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688149 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688152 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688154 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 19:59:31.692231 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688157 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688160 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688162 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688166 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688169 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688172 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688175 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688178 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688180 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688183 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688186 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688188 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688191 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688193 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688196 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688199 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688201 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688204 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688206 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688209 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 19:59:31.692603 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688211 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688214 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688216 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688219 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688221 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688225 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688228 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688231 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688233 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688236 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688238 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688241 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688244 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688246 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688249 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688251 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688254 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688257 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688259 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 19:59:31.693159 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688262 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688264 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688267 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688269 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688272 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688275 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688277 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688280 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688282 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688285 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688287 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688290 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688292 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688295 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688297 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688300 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688302 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688304 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688307 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688309 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 19:59:31.693620 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688312 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688314 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688317 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688319 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688322 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688325 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688327 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688329 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688332 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688335 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688337 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688340 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688342 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688345 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688347 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688350 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688352 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688355 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688357 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688360 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 19:59:31.694108 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688362 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688365 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688367 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688370 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688373 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688376 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:31.688380 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.688385 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.688485 2581 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.690459 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.691992 2581 server.go:1019] "Starting client certificate rotation" Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.692100 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 19:59:31.694574 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.692138 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 19:59:31.694912 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.694893 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 19:59:31.696024 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.696004 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 19:59:31.710409 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.710388 2581 log.go:25] "Validated CRI v1 runtime API" Jul 28 19:59:31.712322 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.712305 2581 log.go:25] "Validated CRI v1 image API" Jul 28 19:59:31.713104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.713089 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 19:59:31.715517 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.715497 2581 fs.go:135] Filesystem UUIDs: map[3d13fb40-ef93-45fd-978e-0e06598df1a5:/dev/nvme0n1p3 73493cc2-6967-4d11-a23b-4d6e6b386bdb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 28 19:59:31.715559 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.715518 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 19:59:31.720420 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.720311 2581 manager.go:216] Machine: {Timestamp:2026-07-28 19:59:31.719755969 +0000 UTC m=+0.338596175 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084787 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cda2c0cafe8c8534e7b5355645001 SystemUUID:ec2cda2c-0caf-e8c8-534e-7b5355645001 BootID:b706d2f9-e660-427a-999d-8513a02d62c7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:1a:2e:94:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:1a:2e:94:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:9e:c5:82:ff:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 19:59:31.720420 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.720419 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 19:59:31.720521 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.720493 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 19:59:31.722848 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.722831 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 19:59:31.723114 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.723084 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 19:59:31.723290 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.723118 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-255.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 19:59:31.724247 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.724236 2581 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 19:59:31.724281 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.724249 2581 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 19:59:31.724281 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.724256 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 19:59:31.725527 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.725514 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 19:59:31.726265 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.726256 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 19:59:31.728185 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728173 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 28 19:59:31.728349 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728339 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 19:59:31.728386 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728371 2581 kubelet.go:486] "Attempting to sync node with API server" Jul 28 19:59:31.728386 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728383 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 19:59:31.728441 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728400 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 19:59:31.728441 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728410 2581 kubelet.go:398] "Adding apiserver pod source" Jul 28 19:59:31.728441 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728426 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 19:59:31.728933 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728919 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 19:59:31.728969 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.728946 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 19:59:31.730965 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.730945 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 19:59:31.732737 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.732723 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 19:59:31.732773 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.732753 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 19:59:31.734887 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734873 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 19:59:31.734937 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734896 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 19:59:31.734937 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734902 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 19:59:31.734937 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734908 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 19:59:31.734937 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734917 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 19:59:31.734937 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734923 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 19:59:31.734937 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734931 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 19:59:31.734937 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734937 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 19:59:31.735119 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734944 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 19:59:31.735119 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734950 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 19:59:31.735119 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734963 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 19:59:31.735119 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.734969 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 19:59:31.735564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.735554 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 19:59:31.735564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.735563 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 19:59:31.739178 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.739164 2581 server.go:1268] "Started kubelet" Jul 28 19:59:31.739268 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.739245 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 19:59:31.739451 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.739427 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 19:59:31.739528 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.739521 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-255.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 19:59:31.739720 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.739690 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 19:59:31.739774 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.739746 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 19:59:31.739950 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.739924 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 19:59:31.741186 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.741164 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 19:59:31.741226 ip-10-0-136-255 systemd[1]: Started Kubernetes Kubelet. Jul 28 19:59:31.743771 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.742904 2581 server.go:310] "Adding debug handlers to kubelet server" Jul 28 19:59:31.746260 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.746238 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 19:59:31.748171 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.747190 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-255.ec2.internal.18c68c7d381bf270 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-255.ec2.internal,UID:ip-10-0-136-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-255.ec2.internal,},FirstTimestamp:2026-07-28 19:59:31.739136624 +0000 UTC m=+0.357976830,LastTimestamp:2026-07-28 19:59:31.739136624 +0000 UTC m=+0.357976830,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-255.ec2.internal,}" Jul 28 19:59:31.749116 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.749100 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qd6jc" Jul 28 19:59:31.749885 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.749870 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 19:59:31.749941 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.749885 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 19:59:31.750068 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.750028 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:31.750481 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750460 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 19:59:31.750481 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750483 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 19:59:31.750671 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750607 2581 reconstruct.go:97] "Volume reconstruction finished" Jul 28 19:59:31.750671 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750616 2581 reconciler.go:29] "Reconciler: start to sync state" Jul 28 19:59:31.750671 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750619 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 19:59:31.750869 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750675 2581 factory.go:55] Registering systemd factory Jul 28 19:59:31.750869 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750690 2581 factory.go:223] Registration of the systemd container factory successfully Jul 28 19:59:31.750974 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750961 2581 factory.go:153] Registering CRI-O factory Jul 28 19:59:31.751057 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.750979 2581 factory.go:223] Registration of the crio container factory successfully Jul 28 19:59:31.751057 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.751031 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 19:59:31.751057 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.751054 2581 factory.go:103] Registering Raw factory Jul 28 19:59:31.751202 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.751068 2581 manager.go:1196] Started watching for new ooms in manager Jul 28 19:59:31.751346 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.751322 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 19:59:31.751526 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.751501 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 19:59:31.751652 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.751639 2581 manager.go:349] Starting recovery of all containers Jul 28 19:59:31.758399 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.757403 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qd6jc" Jul 28 19:59:31.761314 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.761296 2581 manager.go:354] Recovery completed Jul 28 19:59:31.766717 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.766704 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:59:31.768592 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.768576 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:59:31.768657 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.768604 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:59:31.768657 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.768615 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:59:31.769110 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.769092 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 19:59:31.769110 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.769103 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:59:31.769257 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.769108 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 19:59:31.769257 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.769168 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 28 19:59:31.769809 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.769775 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:59:31.769870 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.769827 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:59:31.769870 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.769843 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:59:31.770653 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.770639 2581 policy_none.go:49] "None policy: Start" Jul 28 19:59:31.770653 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.770655 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 19:59:31.770741 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.770665 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 19:59:31.771620 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.771609 2581 policy_none.go:47] "Start" Jul 28 19:59:31.807529 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.807338 2581 manager.go:337] "Starting Device Plugin manager" Jul 28 19:59:31.807651 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.807553 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 19:59:31.807651 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.807564 2581 server.go:87] "Starting device plugin registration server" Jul 28 19:59:31.807837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.807784 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 19:59:31.807885 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.807815 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 19:59:31.807945 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.807915 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 19:59:31.808024 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.808001 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 19:59:31.808024 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:31.808015 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 19:59:31.808556 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.808540 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 19:59:31.808624 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:31.808581 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.082527 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.082500 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:59:33.083761 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.083743 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:59:33.083866 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.083781 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:59:33.083866 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.083795 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:59:33.083866 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.083828 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.092056 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.092039 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.092138 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.092064 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-255.ec2.internal\": node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.107177 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.107129 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.144331 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.144290 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 19:59:33.145459 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.145445 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 19:59:33.145515 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.145484 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 19:59:33.145515 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.145505 2581 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 19:59:33.145575 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.145536 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 19:59:33.147909 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.147892 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:59:33.207436 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.207403 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.245842 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.245809 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal"] Jul 28 19:59:33.245924 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.245892 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:59:33.247521 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.247507 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:59:33.247587 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.247536 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:59:33.247587 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.247547 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:59:33.248877 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.248865 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:59:33.248987 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.248974 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.249023 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.249001 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:59:33.249584 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.249566 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:59:33.249664 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.249589 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:59:33.249664 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.249600 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:59:33.249664 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.249644 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:59:33.249664 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.249662 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:59:33.249825 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.249672 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:59:33.250764 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.250749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.250834 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.250774 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 19:59:33.251434 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.251421 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientMemory" Jul 28 19:59:33.251514 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.251449 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 19:59:33.251514 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.251464 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeHasSufficientPID" Jul 28 19:59:33.287106 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.287084 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-255.ec2.internal\" not found" node="ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.291452 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.291436 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-255.ec2.internal\" not found" node="ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.307850 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.307832 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.326919 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.326895 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf275f86472ebc6b78f28f35a49feca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal\" (UID: \"ebf275f86472ebc6b78f28f35a49feca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.326999 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.326923 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf275f86472ebc6b78f28f35a49feca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal\" (UID: \"ebf275f86472ebc6b78f28f35a49feca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.326999 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.326950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfbeafd1998bc253fc31c2c8fabaa146-config\") pod \"kube-apiserver-proxy-ip-10-0-136-255.ec2.internal\" (UID: \"cfbeafd1998bc253fc31c2c8fabaa146\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.408423 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.408345 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.427703 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.427679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf275f86472ebc6b78f28f35a49feca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal\" (UID: \"ebf275f86472ebc6b78f28f35a49feca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.427765 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.427709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf275f86472ebc6b78f28f35a49feca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal\" (UID: \"ebf275f86472ebc6b78f28f35a49feca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.427765 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.427726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfbeafd1998bc253fc31c2c8fabaa146-config\") pod \"kube-apiserver-proxy-ip-10-0-136-255.ec2.internal\" (UID: \"cfbeafd1998bc253fc31c2c8fabaa146\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.427832 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.427763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cfbeafd1998bc253fc31c2c8fabaa146-config\") pod \"kube-apiserver-proxy-ip-10-0-136-255.ec2.internal\" (UID: \"cfbeafd1998bc253fc31c2c8fabaa146\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.427832 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.427780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf275f86472ebc6b78f28f35a49feca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal\" (UID: \"ebf275f86472ebc6b78f28f35a49feca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.427832 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.427788 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf275f86472ebc6b78f28f35a49feca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal\" (UID: \"ebf275f86472ebc6b78f28f35a49feca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.508862 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.508830 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.589425 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.589387 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.594351 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.594332 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" Jul 28 19:59:33.609576 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.609554 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.710243 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.710147 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.810630 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.810598 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.866195 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.866167 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 19:59:33.866592 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.866302 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 19:59:33.910843 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:33.910816 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:33.922951 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.922928 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:59:33.936596 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.936565 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 19:54:32 +0000 UTC" deadline="2027-12-23 23:07:10.505784082 +0000 UTC" Jul 28 19:59:33.936596 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:33.936593 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12315h7m36.569196325s" Jul 28 19:59:34.011891 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.011813 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:34.112389 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.112361 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:34.213272 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.213240 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:34.251857 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.251830 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:59:34.259049 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:34.259022 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbeafd1998bc253fc31c2c8fabaa146.slice/crio-4976197c9e555caf12e74795aa8ec598e53538e5f16a58758d82e9c1a019bb35 WatchSource:0}: Error finding container 4976197c9e555caf12e74795aa8ec598e53538e5f16a58758d82e9c1a019bb35: Status 404 returned error can't find the container with id 4976197c9e555caf12e74795aa8ec598e53538e5f16a58758d82e9c1a019bb35 Jul 28 19:59:34.259424 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:34.259402 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf275f86472ebc6b78f28f35a49feca.slice/crio-52e7d4057e4b24b26a1a8dd62b88bfc443efbeaecc985729b64cc4d26a5adf75 WatchSource:0}: Error finding container 52e7d4057e4b24b26a1a8dd62b88bfc443efbeaecc985729b64cc4d26a5adf75: Status 404 returned error can't find the container with id 52e7d4057e4b24b26a1a8dd62b88bfc443efbeaecc985729b64cc4d26a5adf75 Jul 28 19:59:34.263561 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.263546 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 19:59:34.313539 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.313504 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:34.414281 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.414255 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-255.ec2.internal\" not found" Jul 28 19:59:34.460672 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.460645 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:59:34.525515 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.525442 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" Jul 28 19:59:34.537877 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.537855 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 19:59:34.539258 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.539244 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" Jul 28 19:59:34.547121 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.547103 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 19:59:34.904350 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.904028 2581 apiserver.go:52] "Watching apiserver" Jul 28 19:59:34.909000 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.908976 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 19:59:34.915685 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.915651 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-mss6g","openshift-ovn-kubernetes/ovnkube-node-kk9q6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn","openshift-cluster-node-tuning-operator/tuned-knj5g","openshift-image-registry/node-ca-5vfv9","openshift-multus/multus-additional-cni-plugins-nmjk4","openshift-multus/multus-v22pp","openshift-multus/network-metrics-daemon-xzqcj","kube-system/global-pull-secret-syncer-sq8gz","kube-system/konnectivity-agent-z4rd8","kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal","openshift-dns/node-resolver-7mnmq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal","openshift-network-diagnostics/network-check-target-vk459"] Jul 28 19:59:34.917379 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.917354 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.918764 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.918508 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:34.918764 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.918579 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:34.920391 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.919249 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 19:59:34.920391 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.919490 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 19:59:34.920391 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.919741 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fcfqc\"" Jul 28 19:59:34.920391 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.919903 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 19:59:34.920391 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.920090 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 19:59:34.922563 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.922237 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.923664 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.923646 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.923798 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.923779 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 19:59:34.923942 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.923927 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6699\"" Jul 28 19:59:34.924189 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.924168 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 19:59:34.924371 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.924347 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 19:59:34.924748 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.924728 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 19:59:34.924948 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.924931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:34.925394 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.925376 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:59:34.925691 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.925674 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlsrq\"" Jul 28 19:59:34.925775 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.925688 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 19:59:34.926510 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.926492 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 19:59:34.926932 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.926915 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7nw7\"" Jul 28 19:59:34.927160 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.927144 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 19:59:34.927880 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.927372 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 19:59:34.929437 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.929421 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.929437 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.929431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:34.929785 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.929762 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:34.930634 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.930614 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:34.931592 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.931576 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nnsqg\"" Jul 28 19:59:34.931977 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.931779 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 19:59:34.931977 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.931796 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 19:59:34.932175 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.932155 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:34.932720 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.932699 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h7694\"" Jul 28 19:59:34.932720 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.932716 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 19:59:34.932856 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.932782 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 19:59:34.932856 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.932800 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 19:59:34.933856 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.933836 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 19:59:34.934506 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.934097 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 19:59:34.934506 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.934110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:34.935006 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.934988 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 19:59:34.936063 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.935473 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbl52\"" Jul 28 19:59:34.936209 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.935526 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 19:59:34.936394 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.935583 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 19:59:34.936582 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.935713 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 19:59:34.936582 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936008 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-cnibin\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936840 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-os-release\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936891 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-netns\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936916 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-kubelet\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936944 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysconfig\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-kubernetes\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.936987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-host\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-socket-dir-parent\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-k8s-cni-cncf-io\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-sys\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937100 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-lib-modules\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937138 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-var-lib-kubelet\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-serviceca\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:34.937560 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937210 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-system-cni-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937222 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s9j25\"" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937242 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937278 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-cni-bin\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-cni-binary-copy\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-hostroot\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-daemon-config\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-etc-kubernetes\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxx5h\" (UniqueName: \"kubernetes.io/projected/6bca7392-4c0d-4e95-9e7c-3eb68914d111-kube-api-access-lxx5h\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937570 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-registration-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-sys-fs\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937612 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lchk\" (UniqueName: \"kubernetes.io/projected/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-kube-api-access-9lchk\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-cni-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-cnibin\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.938343 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-conf-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-multus-certs\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfhl\" (UniqueName: \"kubernetes.io/projected/099719d8-3d25-418e-b055-a4a1363e2765-kube-api-access-4cfhl\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysctl-d\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937782 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62tn\" (UniqueName: \"kubernetes.io/projected/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-kube-api-access-b62tn\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-socket-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937849 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-etc-selinux\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b5b556cf-24a3-4fcf-a764-d2f748624105-kube-api-access-kvj4g\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-cni-multus\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.937980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-device-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938007 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysctl-conf\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-run\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-host\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-tuned\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938089 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdfc0ce9-10c1-489b-acbf-21e213c48a11-tmp\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939161 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-system-cni-dir\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:34.939984 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-os-release\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:34.939984 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938202 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-modprobe-d\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939984 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-systemd\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939984 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwmcl\" (UniqueName: \"kubernetes.io/projected/fdfc0ce9-10c1-489b-acbf-21e213c48a11-kube-api-access-gwmcl\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:34.939984 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.938391 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:34.939984 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:34.938448 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:34.944688 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.944667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:34.946400 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.946384 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 19:59:34.946761 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.946743 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 19:59:34.946992 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.946502 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lzwf8\"" Jul 28 19:59:34.958120 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.958087 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8fxkz" Jul 28 19:59:34.965305 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.965282 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8fxkz" Jul 28 19:59:34.981148 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:34.981120 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 19:59:35.026170 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.026145 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 19:59:35.039064 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039032 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62tn\" (UniqueName: \"kubernetes.io/projected/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-kube-api-access-b62tn\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.039233 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039074 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-etc-selinux\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.039233 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039130 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-run-netns\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.039233 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039165 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-env-overrides\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.039233 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f668389-c6ba-44b7-aded-ab82eb3cc122-kubelet-config\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.039233 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039212 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-etc-selinux\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.039233 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-cni-multus\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-device-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039311 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039368 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-ovnkube-script-lib\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-socket-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysctl-conf\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-tuned\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-os-release\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-cni-bin\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfkk\" (UniqueName: \"kubernetes.io/projected/ab6059fd-4193-477e-b5f1-7dc7da431cb2-kube-api-access-rnfkk\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxx5h\" (UniqueName: \"kubernetes.io/projected/6bca7392-4c0d-4e95-9e7c-3eb68914d111-kube-api-access-lxx5h\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039561 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-socket-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.039585 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-modprobe-d\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-cnibin\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039604 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-cni-multus\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-os-release\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039681 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-cnibin\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039699 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-device-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039815 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-modprobe-d\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039833 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysctl-conf\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.039947 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-os-release\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-ovn\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040089 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28be3705-4916-4862-bd49-84caadef8558-ovn-node-metrics-cert\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.040162 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfhl\" (UniqueName: \"kubernetes.io/projected/099719d8-3d25-418e-b055-a4a1363e2765-kube-api-access-4cfhl\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-os-release\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-kubernetes\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-host\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040496 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-kubernetes\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040509 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040532 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-host\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-socket-dir-parent\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040591 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-k8s-cni-cncf-io\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-node-log\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040635 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-socket-dir-parent\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040646 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-sys\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-lib-modules\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040711 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-k8s-cni-cncf-io\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040723 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.040745 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040749 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-sys\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-serviceca\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-lib-modules\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-system-cni-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040905 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-system-cni-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-cni-bin\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040981 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-cni-bin\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.040990 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-slash\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-cni-binary-copy\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041056 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-etc-kubernetes\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.041072 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-run-ovn-kubernetes\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041109 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3319a3-c972-4ea8-9a22-7faa5baed072-agent-certs\") pod \"konnectivity-agent-z4rd8\" (UID: \"ee3319a3-c972-4ea8-9a22-7faa5baed072\") " pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.041159 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs podName:6bca7392-4c0d-4e95-9e7c-3eb68914d111 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:35.541118937 +0000 UTC m=+2.985429950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs") pod "network-metrics-daemon-xzqcj" (UID: "6bca7392-4c0d-4e95-9e7c-3eb68914d111") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-cni-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.041288 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-cnibin\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-conf-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-ovnkube-config\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041375 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysctl-d\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041450 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-etc-kubernetes\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041464 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041483 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-var-lib-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041558 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysctl-d\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041586 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-cni-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-etc-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041647 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-log-socket\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041654 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-conf-dir\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041658 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-cni-binary-copy\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042104 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96c649c0-feb1-43e1-98d5-c31848ac7b64-hosts-file\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041224 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-serviceca\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041708 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-cnibin\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b5b556cf-24a3-4fcf-a764-d2f748624105-kube-api-access-kvj4g\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqx9\" (UniqueName: \"kubernetes.io/projected/96c649c0-feb1-43e1-98d5-c31848ac7b64-kube-api-access-mpqx9\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-kubelet\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041778 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-cni-binary-copy\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041788 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-run\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041816 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-host\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041846 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdfc0ce9-10c1-489b-acbf-21e213c48a11-tmp\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041852 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-run\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-system-cni-dir\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-kubelet\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f668389-c6ba-44b7-aded-ab82eb3cc122-dbus\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-host\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-daemon-config\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041966 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5b556cf-24a3-4fcf-a764-d2f748624105-system-cni-dir\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.042837 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.041978 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-var-lib-kubelet\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042001 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-registration-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042025 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-sys-fs\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-systemd\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042103 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-sys-fs\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042111 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-systemd\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042104 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/099719d8-3d25-418e-b055-a4a1363e2765-registration-dir\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwmcl\" (UniqueName: \"kubernetes.io/projected/fdfc0ce9-10c1-489b-acbf-21e213c48a11-kube-api-access-gwmcl\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042176 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-netns\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042228 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysconfig\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3319a3-c972-4ea8-9a22-7faa5baed072-konnectivity-ca\") pod \"konnectivity-agent-z4rd8\" (UID: \"ee3319a3-c972-4ea8-9a22-7faa5baed072\") " pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042243 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-netns\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-systemd-units\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-sysconfig\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.043682 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-var-lib-kubelet\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042347 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2s88\" (UniqueName: \"kubernetes.io/projected/28be3705-4916-4862-bd49-84caadef8558-kube-api-access-x2s88\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042372 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab6059fd-4193-477e-b5f1-7dc7da431cb2-iptables-alerter-script\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab6059fd-4193-477e-b5f1-7dc7da431cb2-host-slash\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-hostroot\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-systemd\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042446 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdfc0ce9-10c1-489b-acbf-21e213c48a11-var-lib-kubelet\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-cni-netd\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042510 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-hostroot\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042520 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c649c0-feb1-43e1-98d5-c31848ac7b64-tmp-dir\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lchk\" (UniqueName: \"kubernetes.io/projected/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-kube-api-access-9lchk\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042561 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-multus-daemon-config\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-multus-certs\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042647 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5b556cf-24a3-4fcf-a764-d2f748624105-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.042725 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-host-run-multus-certs\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.044539 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.043763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdfc0ce9-10c1-489b-acbf-21e213c48a11-etc-tuned\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.045260 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.044900 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdfc0ce9-10c1-489b-acbf-21e213c48a11-tmp\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.046350 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.046309 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62tn\" (UniqueName: \"kubernetes.io/projected/e9090d6e-910d-40e6-8fee-7efa6e20cbe2-kube-api-access-b62tn\") pod \"multus-v22pp\" (UID: \"e9090d6e-910d-40e6-8fee-7efa6e20cbe2\") " pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.049925 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.049896 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfhl\" (UniqueName: \"kubernetes.io/projected/099719d8-3d25-418e-b055-a4a1363e2765-kube-api-access-4cfhl\") pod \"aws-ebs-csi-driver-node-cn4cn\" (UID: \"099719d8-3d25-418e-b055-a4a1363e2765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.054554 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.053737 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lchk\" (UniqueName: \"kubernetes.io/projected/c4e9a8ae-a566-49ae-a090-aae3e55b73e9-kube-api-access-9lchk\") pod \"node-ca-5vfv9\" (UID: \"c4e9a8ae-a566-49ae-a090-aae3e55b73e9\") " pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:35.054554 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.053869 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwmcl\" (UniqueName: \"kubernetes.io/projected/fdfc0ce9-10c1-489b-acbf-21e213c48a11-kube-api-access-gwmcl\") pod \"tuned-knj5g\" (UID: \"fdfc0ce9-10c1-489b-acbf-21e213c48a11\") " pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.057194 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.057155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxx5h\" (UniqueName: \"kubernetes.io/projected/6bca7392-4c0d-4e95-9e7c-3eb68914d111-kube-api-access-lxx5h\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:35.058509 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.058486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b5b556cf-24a3-4fcf-a764-d2f748624105-kube-api-access-kvj4g\") pod \"multus-additional-cni-plugins-nmjk4\" (UID: \"b5b556cf-24a3-4fcf-a764-d2f748624105\") " pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.143172 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab6059fd-4193-477e-b5f1-7dc7da431cb2-iptables-alerter-script\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.143338 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab6059fd-4193-477e-b5f1-7dc7da431cb2-host-slash\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.143338 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-systemd\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143338 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143253 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-cni-netd\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143338 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c649c0-feb1-43e1-98d5-c31848ac7b64-tmp-dir\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.143564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-run-netns\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-env-overrides\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143408 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f668389-c6ba-44b7-aded-ab82eb3cc122-kubelet-config\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.143564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143434 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-ovnkube-script-lib\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-cni-bin\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143564 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfkk\" (UniqueName: \"kubernetes.io/projected/ab6059fd-4193-477e-b5f1-7dc7da431cb2-kube-api-access-rnfkk\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-ovn\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143640 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28be3705-4916-4862-bd49-84caadef8558-ovn-node-metrics-cert\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143719 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-node-log\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143738 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab6059fd-4193-477e-b5f1-7dc7da431cb2-iptables-alerter-script\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-slash\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-run-ovn-kubernetes\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143820 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8f668389-c6ba-44b7-aded-ab82eb3cc122-kubelet-config\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3319a3-c972-4ea8-9a22-7faa5baed072-agent-certs\") pod \"konnectivity-agent-z4rd8\" (UID: \"ee3319a3-c972-4ea8-9a22-7faa5baed072\") " pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143859 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143865 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab6059fd-4193-477e-b5f1-7dc7da431cb2-host-slash\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143886 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-ovnkube-config\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.143904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143909 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-systemd\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143948 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-cni-netd\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.143979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-var-lib-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-etc-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-log-socket\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96c649c0-feb1-43e1-98d5-c31848ac7b64-hosts-file\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144139 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqx9\" (UniqueName: \"kubernetes.io/projected/96c649c0-feb1-43e1-98d5-c31848ac7b64-kube-api-access-mpqx9\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-kubelet\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f668389-c6ba-44b7-aded-ab82eb3cc122-dbus\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144211 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96c649c0-feb1-43e1-98d5-c31848ac7b64-tmp-dir\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3319a3-c972-4ea8-9a22-7faa5baed072-konnectivity-ca\") pod \"konnectivity-agent-z4rd8\" (UID: \"ee3319a3-c972-4ea8-9a22-7faa5baed072\") " pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144275 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-systemd-units\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144276 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8f668389-c6ba-44b7-aded-ab82eb3cc122-dbus\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144312 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144348 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-run-netns\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.144547 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144357 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2s88\" (UniqueName: \"kubernetes.io/projected/28be3705-4916-4862-bd49-84caadef8558-kube-api-access-x2s88\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145358 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.144769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-env-overrides\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145358 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145083 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-node-log\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145358 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145113 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-ovnkube-script-lib\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145358 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145140 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-cni-bin\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145358 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145186 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-run-ovn\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145358 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.145204 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:35.145358 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.145271 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret podName:8f668389-c6ba-44b7-aded-ab82eb3cc122 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:35.645253944 +0000 UTC m=+3.089564954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret") pod "global-pull-secret-syncer-sq8gz" (UID: "8f668389-c6ba-44b7-aded-ab82eb3cc122") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145390 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-systemd-units\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145454 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-run-ovn-kubernetes\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145468 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28be3705-4916-4862-bd49-84caadef8558-ovnkube-config\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145527 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-slash\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145590 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96c649c0-feb1-43e1-98d5-c31848ac7b64-hosts-file\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145612 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-etc-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145635 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-log-socket\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145677 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-var-lib-openvswitch\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145681 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.145762 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.145710 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28be3705-4916-4862-bd49-84caadef8558-host-kubelet\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.147709 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.147682 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee3319a3-c972-4ea8-9a22-7faa5baed072-konnectivity-ca\") pod \"konnectivity-agent-z4rd8\" (UID: \"ee3319a3-c972-4ea8-9a22-7faa5baed072\") " pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:35.148274 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.148087 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28be3705-4916-4862-bd49-84caadef8558-ovn-node-metrics-cert\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.148542 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.148525 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee3319a3-c972-4ea8-9a22-7faa5baed072-agent-certs\") pod \"konnectivity-agent-z4rd8\" (UID: \"ee3319a3-c972-4ea8-9a22-7faa5baed072\") " pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:35.151893 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.151693 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:59:35.151893 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.151715 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:59:35.151893 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.151729 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nwlmf for pod openshift-network-diagnostics/network-check-target-vk459: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:35.151893 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.151801 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf podName:f60c1e92-7837-41b8-9bf1-0bebd43c48c7 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:35.651782941 +0000 UTC m=+3.096093953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nwlmf" (UniqueName: "kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf") pod "network-check-target-vk459" (UID: "f60c1e92-7837-41b8-9bf1-0bebd43c48c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:35.152567 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.152513 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" event={"ID":"cfbeafd1998bc253fc31c2c8fabaa146","Type":"ContainerStarted","Data":"4976197c9e555caf12e74795aa8ec598e53538e5f16a58758d82e9c1a019bb35"} Jul 28 19:59:35.154904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.154784 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" event={"ID":"ebf275f86472ebc6b78f28f35a49feca","Type":"ContainerStarted","Data":"52e7d4057e4b24b26a1a8dd62b88bfc443efbeaecc985729b64cc4d26a5adf75"} Jul 28 19:59:35.155565 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.155389 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfkk\" (UniqueName: \"kubernetes.io/projected/ab6059fd-4193-477e-b5f1-7dc7da431cb2-kube-api-access-rnfkk\") pod \"iptables-alerter-mss6g\" (UID: \"ab6059fd-4193-477e-b5f1-7dc7da431cb2\") " pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.155565 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.155491 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqx9\" (UniqueName: \"kubernetes.io/projected/96c649c0-feb1-43e1-98d5-c31848ac7b64-kube-api-access-mpqx9\") pod \"node-resolver-7mnmq\" (UID: \"96c649c0-feb1-43e1-98d5-c31848ac7b64\") " pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.156805 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.156783 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2s88\" (UniqueName: \"kubernetes.io/projected/28be3705-4916-4862-bd49-84caadef8558-kube-api-access-x2s88\") pod \"ovnkube-node-kk9q6\" (UID: \"28be3705-4916-4862-bd49-84caadef8558\") " pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.231042 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.231015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v22pp" Jul 28 19:59:35.246079 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.246046 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" Jul 28 19:59:35.257042 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.256961 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-knj5g" Jul 28 19:59:35.265676 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.265655 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vfv9" Jul 28 19:59:35.272661 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.272639 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" Jul 28 19:59:35.283208 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.283184 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mss6g" Jul 28 19:59:35.292852 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.292832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 19:59:35.299423 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.299396 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:35.305974 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.305956 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7mnmq" Jul 28 19:59:35.546950 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.546921 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:35.547100 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.547073 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:35.547142 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.547137 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs podName:6bca7392-4c0d-4e95-9e7c-3eb68914d111 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:36.547120179 +0000 UTC m=+3.991431173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs") pod "network-metrics-daemon-xzqcj" (UID: "6bca7392-4c0d-4e95-9e7c-3eb68914d111") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:35.647237 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.647202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:35.647422 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.647391 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:35.647479 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.647463 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret podName:8f668389-c6ba-44b7-aded-ab82eb3cc122 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:36.647443809 +0000 UTC m=+4.091754806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret") pod "global-pull-secret-syncer-sq8gz" (UID: "8f668389-c6ba-44b7-aded-ab82eb3cc122") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:35.747980 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.747946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:35.748153 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.748099 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:59:35.748153 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.748120 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:59:35.748153 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.748131 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nwlmf for pod openshift-network-diagnostics/network-check-target-vk459: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:35.748264 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:35.748187 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf podName:f60c1e92-7837-41b8-9bf1-0bebd43c48c7 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:36.748169587 +0000 UTC m=+4.192480597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwlmf" (UniqueName: "kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf") pod "network-check-target-vk459" (UID: "f60c1e92-7837-41b8-9bf1-0bebd43c48c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:35.969071 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.968984 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:54:34 +0000 UTC" deadline="2027-12-27 11:21:25.126257938 +0000 UTC" Jul 28 19:59:35.969071 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:35.969023 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12399h21m49.157238635s" Jul 28 19:59:36.146512 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:36.146480 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:36.146659 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.146612 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:36.186152 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.186124 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9090d6e_910d_40e6_8fee_7efa6e20cbe2.slice/crio-96178bf69b0610c3751ce18f3b3a25be44bd4c85ca789dc64d7548e308a40498 WatchSource:0}: Error finding container 96178bf69b0610c3751ce18f3b3a25be44bd4c85ca789dc64d7548e308a40498: Status 404 returned error can't find the container with id 96178bf69b0610c3751ce18f3b3a25be44bd4c85ca789dc64d7548e308a40498 Jul 28 19:59:36.202188 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.202124 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b556cf_24a3_4fcf_a764_d2f748624105.slice/crio-f14659bece13471bcb78758a7da5888ac6495c11a2fb8dea654f2d3763513056 WatchSource:0}: Error finding container f14659bece13471bcb78758a7da5888ac6495c11a2fb8dea654f2d3763513056: Status 404 returned error can't find the container with id f14659bece13471bcb78758a7da5888ac6495c11a2fb8dea654f2d3763513056 Jul 28 19:59:36.203190 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.203165 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c649c0_feb1_43e1_98d5_c31848ac7b64.slice/crio-d5cdc09d10312e454e6e1ba9cbd9ef2939b8e6ce70b49542917432853f816460 WatchSource:0}: Error finding container d5cdc09d10312e454e6e1ba9cbd9ef2939b8e6ce70b49542917432853f816460: Status 404 returned error can't find the container with id d5cdc09d10312e454e6e1ba9cbd9ef2939b8e6ce70b49542917432853f816460 Jul 28 19:59:36.204423 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.204172 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e9a8ae_a566_49ae_a090_aae3e55b73e9.slice/crio-cb3ef3d00900ebec29f96f4c03fb2635dec9eb24e14b2760a20ceb528175f2a4 WatchSource:0}: Error finding container cb3ef3d00900ebec29f96f4c03fb2635dec9eb24e14b2760a20ceb528175f2a4: Status 404 returned error can't find the container with id cb3ef3d00900ebec29f96f4c03fb2635dec9eb24e14b2760a20ceb528175f2a4 Jul 28 19:59:36.205984 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.205849 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3319a3_c972_4ea8_9a22_7faa5baed072.slice/crio-bb6583a8f04a4feb099f93f93cc4d22b6bcd7a5bf9dd0ccb9cc57432ba68163c WatchSource:0}: Error finding container bb6583a8f04a4feb099f93f93cc4d22b6bcd7a5bf9dd0ccb9cc57432ba68163c: Status 404 returned error can't find the container with id bb6583a8f04a4feb099f93f93cc4d22b6bcd7a5bf9dd0ccb9cc57432ba68163c Jul 28 19:59:36.206563 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.206538 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099719d8_3d25_418e_b055_a4a1363e2765.slice/crio-990002364e86a05a8f5cba2c455e5b43f25a6fdf46765b61161858ab00eb1be6 WatchSource:0}: Error finding container 990002364e86a05a8f5cba2c455e5b43f25a6fdf46765b61161858ab00eb1be6: Status 404 returned error can't find the container with id 990002364e86a05a8f5cba2c455e5b43f25a6fdf46765b61161858ab00eb1be6 Jul 28 19:59:36.207785 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.207763 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfc0ce9_10c1_489b_acbf_21e213c48a11.slice/crio-4e2a38b7eca461296b592218c2e1751942bbb72f09f39b4d708e468dc74b1d9a WatchSource:0}: Error finding container 4e2a38b7eca461296b592218c2e1751942bbb72f09f39b4d708e468dc74b1d9a: Status 404 returned error can't find the container with id 4e2a38b7eca461296b592218c2e1751942bbb72f09f39b4d708e468dc74b1d9a Jul 28 19:59:36.209155 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.208902 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28be3705_4916_4862_bd49_84caadef8558.slice/crio-cb915818bff0b9281e35ca00d50c3b6e12e3532c15c408e1171e5aa6f058ee15 WatchSource:0}: Error finding container cb915818bff0b9281e35ca00d50c3b6e12e3532c15c408e1171e5aa6f058ee15: Status 404 returned error can't find the container with id cb915818bff0b9281e35ca00d50c3b6e12e3532c15c408e1171e5aa6f058ee15 Jul 28 19:59:36.210425 ip-10-0-136-255 kubenswrapper[2581]: W0728 19:59:36.210276 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6059fd_4193_477e_b5f1_7dc7da431cb2.slice/crio-39a922cf1be49c869bff00173180b4cc71e123b993ed3f0388b8dec5de95764f WatchSource:0}: Error finding container 39a922cf1be49c869bff00173180b4cc71e123b993ed3f0388b8dec5de95764f: Status 404 returned error can't find the container with id 39a922cf1be49c869bff00173180b4cc71e123b993ed3f0388b8dec5de95764f Jul 28 19:59:36.553890 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:36.553860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:36.554062 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.553985 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:36.554062 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.554046 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs podName:6bca7392-4c0d-4e95-9e7c-3eb68914d111 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:38.554026684 +0000 UTC m=+5.998337678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs") pod "network-metrics-daemon-xzqcj" (UID: "6bca7392-4c0d-4e95-9e7c-3eb68914d111") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:36.655222 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:36.655185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:36.655427 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.655410 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:36.655493 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.655480 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret podName:8f668389-c6ba-44b7-aded-ab82eb3cc122 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:38.655460345 +0000 UTC m=+6.099771341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret") pod "global-pull-secret-syncer-sq8gz" (UID: "8f668389-c6ba-44b7-aded-ab82eb3cc122") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:36.756517 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:36.755865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:36.756517 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.756057 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:59:36.756517 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.756082 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:59:36.756517 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.756094 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nwlmf for pod openshift-network-diagnostics/network-check-target-vk459: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:36.756517 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:36.756158 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf podName:f60c1e92-7837-41b8-9bf1-0bebd43c48c7 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:38.756137862 +0000 UTC m=+6.200448859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwlmf" (UniqueName: "kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf") pod "network-check-target-vk459" (UID: "f60c1e92-7837-41b8-9bf1-0bebd43c48c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:36.969979 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:36.969770 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:54:34 +0000 UTC" deadline="2028-03-19 00:23:31.801892478 +0000 UTC" Jul 28 19:59:36.969979 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:36.969833 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14380h23m54.832063395s" Jul 28 19:59:37.148797 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.148766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:37.148947 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:37.148916 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:37.149014 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.148998 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:37.149116 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:37.149098 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:37.174024 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.173988 2581 generic.go:356] "Generic (PLEG): container finished" podID="ebf275f86472ebc6b78f28f35a49feca" containerID="5f6d1a86d12f3378e41995883bfc0b620c0ceae45ec0418d02786452b1155f75" exitCode=0 Jul 28 19:59:37.174421 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.174350 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" event={"ID":"ebf275f86472ebc6b78f28f35a49feca","Type":"ContainerDied","Data":"5f6d1a86d12f3378e41995883bfc0b620c0ceae45ec0418d02786452b1155f75"} Jul 28 19:59:37.176994 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.176900 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z4rd8" event={"ID":"ee3319a3-c972-4ea8-9a22-7faa5baed072","Type":"ContainerStarted","Data":"bb6583a8f04a4feb099f93f93cc4d22b6bcd7a5bf9dd0ccb9cc57432ba68163c"} Jul 28 19:59:37.186238 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.186187 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerStarted","Data":"f14659bece13471bcb78758a7da5888ac6495c11a2fb8dea654f2d3763513056"} Jul 28 19:59:37.190904 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.190880 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v22pp" event={"ID":"e9090d6e-910d-40e6-8fee-7efa6e20cbe2","Type":"ContainerStarted","Data":"96178bf69b0610c3751ce18f3b3a25be44bd4c85ca789dc64d7548e308a40498"} Jul 28 19:59:37.196108 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.196064 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mss6g" event={"ID":"ab6059fd-4193-477e-b5f1-7dc7da431cb2","Type":"ContainerStarted","Data":"39a922cf1be49c869bff00173180b4cc71e123b993ed3f0388b8dec5de95764f"} Jul 28 19:59:37.204696 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.204668 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mnmq" event={"ID":"96c649c0-feb1-43e1-98d5-c31848ac7b64","Type":"ContainerStarted","Data":"d5cdc09d10312e454e6e1ba9cbd9ef2939b8e6ce70b49542917432853f816460"} Jul 28 19:59:37.209153 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.209126 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"cb915818bff0b9281e35ca00d50c3b6e12e3532c15c408e1171e5aa6f058ee15"} Jul 28 19:59:37.213626 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.213598 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-knj5g" event={"ID":"fdfc0ce9-10c1-489b-acbf-21e213c48a11","Type":"ContainerStarted","Data":"4e2a38b7eca461296b592218c2e1751942bbb72f09f39b4d708e468dc74b1d9a"} Jul 28 19:59:37.225080 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.225048 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" event={"ID":"099719d8-3d25-418e-b055-a4a1363e2765","Type":"ContainerStarted","Data":"990002364e86a05a8f5cba2c455e5b43f25a6fdf46765b61161858ab00eb1be6"} Jul 28 19:59:37.233568 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.233539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vfv9" event={"ID":"c4e9a8ae-a566-49ae-a090-aae3e55b73e9","Type":"ContainerStarted","Data":"cb3ef3d00900ebec29f96f4c03fb2635dec9eb24e14b2760a20ceb528175f2a4"} Jul 28 19:59:37.246227 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:37.246149 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" event={"ID":"cfbeafd1998bc253fc31c2c8fabaa146","Type":"ContainerStarted","Data":"612ecf349132f8fbd7b64d17268692849cf8d77fb0002c116aa75ba75121c9cf"} Jul 28 19:59:38.146743 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:38.146225 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:38.146743 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.146371 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:38.274519 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:38.274389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" event={"ID":"ebf275f86472ebc6b78f28f35a49feca","Type":"ContainerStarted","Data":"4b26f7c4580707cb21462b42e6403d975e6e17e3725533867511a1fb068ebde6"} Jul 28 19:59:38.288389 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:38.287489 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-255.ec2.internal" podStartSLOduration=4.287465675 podStartE2EDuration="4.287465675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:37.26168618 +0000 UTC m=+4.705997197" watchObservedRunningTime="2026-07-28 19:59:38.287465675 +0000 UTC m=+5.731776693" Jul 28 19:59:38.573563 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:38.573512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:38.573756 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.573722 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:38.573818 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.573794 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs podName:6bca7392-4c0d-4e95-9e7c-3eb68914d111 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:42.573773961 +0000 UTC m=+10.018084978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs") pod "network-metrics-daemon-xzqcj" (UID: "6bca7392-4c0d-4e95-9e7c-3eb68914d111") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:38.674767 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:38.674728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:38.674941 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.674926 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:38.675013 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.674992 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret podName:8f668389-c6ba-44b7-aded-ab82eb3cc122 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:42.674974816 +0000 UTC m=+10.119285813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret") pod "global-pull-secret-syncer-sq8gz" (UID: "8f668389-c6ba-44b7-aded-ab82eb3cc122") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:38.776600 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:38.775957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:38.776600 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.776137 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:59:38.776600 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.776156 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:59:38.776600 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.776169 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nwlmf for pod openshift-network-diagnostics/network-check-target-vk459: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:38.776600 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:38.776229 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf podName:f60c1e92-7837-41b8-9bf1-0bebd43c48c7 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:42.776211447 +0000 UTC m=+10.220522454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwlmf" (UniqueName: "kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf") pod "network-check-target-vk459" (UID: "f60c1e92-7837-41b8-9bf1-0bebd43c48c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:39.148481 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:39.148436 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:39.148984 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:39.148594 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:39.148984 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:39.148976 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:39.149094 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:39.149073 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:40.145972 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:40.145932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:40.146163 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:40.146085 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:41.145944 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:41.145911 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:41.146437 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:41.146055 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:41.146437 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:41.146372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:41.146569 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:41.146489 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:42.146077 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:42.146043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:42.146552 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.146179 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:42.610645 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:42.610053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:42.610645 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.610229 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:42.610645 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.610294 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs podName:6bca7392-4c0d-4e95-9e7c-3eb68914d111 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:50.610274372 +0000 UTC m=+18.054585368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs") pod "network-metrics-daemon-xzqcj" (UID: "6bca7392-4c0d-4e95-9e7c-3eb68914d111") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:42.710679 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:42.710641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:42.710873 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.710821 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:42.710934 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.710903 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret podName:8f668389-c6ba-44b7-aded-ab82eb3cc122 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:50.710883783 +0000 UTC m=+18.155194785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret") pod "global-pull-secret-syncer-sq8gz" (UID: "8f668389-c6ba-44b7-aded-ab82eb3cc122") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:42.811821 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:42.811568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:42.811821 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.811790 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:59:42.811821 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.811815 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:59:42.811821 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.811827 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nwlmf for pod openshift-network-diagnostics/network-check-target-vk459: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:42.812136 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:42.811889 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf podName:f60c1e92-7837-41b8-9bf1-0bebd43c48c7 nodeName:}" failed. No retries permitted until 2026-07-28 19:59:50.811872555 +0000 UTC m=+18.256183566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwlmf" (UniqueName: "kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf") pod "network-check-target-vk459" (UID: "f60c1e92-7837-41b8-9bf1-0bebd43c48c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:43.147798 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:43.147356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:43.147798 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:43.147384 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:43.147798 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:43.147468 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:43.147798 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:43.147602 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:44.146835 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:44.146799 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:44.147006 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:44.146944 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:45.145996 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:45.145957 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:45.146494 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:45.146092 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:45.146572 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:45.146500 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:45.146675 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:45.146652 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:46.146294 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:46.146256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:46.146695 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:46.146399 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:47.146100 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:47.146014 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:47.146100 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:47.146059 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:47.146344 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:47.146164 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:47.146344 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:47.146279 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:48.146444 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:48.146408 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:48.146910 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:48.146546 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:49.146208 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:49.146179 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:49.146414 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:49.146223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:49.146414 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:49.146332 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:49.146846 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:49.146414 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:50.145942 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:50.145904 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:50.146189 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.146029 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:50.671378 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:50.671301 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:50.671911 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.671463 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:50.671911 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.671539 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs podName:6bca7392-4c0d-4e95-9e7c-3eb68914d111 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:06.671518718 +0000 UTC m=+34.115829714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs") pod "network-metrics-daemon-xzqcj" (UID: "6bca7392-4c0d-4e95-9e7c-3eb68914d111") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 19:59:50.772173 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:50.772137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:50.772387 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.772287 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:50.772387 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.772372 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret podName:8f668389-c6ba-44b7-aded-ab82eb3cc122 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:06.772353383 +0000 UTC m=+34.216664377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret") pod "global-pull-secret-syncer-sq8gz" (UID: "8f668389-c6ba-44b7-aded-ab82eb3cc122") : object "kube-system"/"original-pull-secret" not registered Jul 28 19:59:50.873149 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:50.873111 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:50.873349 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.873265 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 19:59:50.873349 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.873281 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 19:59:50.873349 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.873293 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nwlmf for pod openshift-network-diagnostics/network-check-target-vk459: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:50.873486 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:50.873378 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf podName:f60c1e92-7837-41b8-9bf1-0bebd43c48c7 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:06.873359144 +0000 UTC m=+34.317670158 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwlmf" (UniqueName: "kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf") pod "network-check-target-vk459" (UID: "f60c1e92-7837-41b8-9bf1-0bebd43c48c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 19:59:51.146106 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:51.146074 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:51.146299 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:51.146116 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:51.146299 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:51.146201 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:51.146434 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:51.146360 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:52.146520 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:52.146483 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:52.146910 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:52.146603 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:53.146874 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:53.146835 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:53.147292 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:53.146970 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:53.147292 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:53.147019 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:53.147292 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:53.147182 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:54.146097 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:54.146060 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:54.146294 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:54.146189 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:55.146111 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.145947 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:55.146732 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.146005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:55.146732 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:55.146188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:55.146732 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:55.146262 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:55.311492 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.311155 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z4rd8" event={"ID":"ee3319a3-c972-4ea8-9a22-7faa5baed072","Type":"ContainerStarted","Data":"d1a1dc75800788fb5d46ccdb2ecc67cc3e234616b8149aebb6cbc38f20023b37"} Jul 28 19:59:55.313266 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.313175 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5b556cf-24a3-4fcf-a764-d2f748624105" containerID="5b207c14d706e2625fa002eb03cc30e93cd4f07f37c3e9b01f96016b60e27530" exitCode=0 Jul 28 19:59:55.313266 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.313231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerDied","Data":"5b207c14d706e2625fa002eb03cc30e93cd4f07f37c3e9b01f96016b60e27530"} Jul 28 19:59:55.315729 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.315691 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v22pp" event={"ID":"e9090d6e-910d-40e6-8fee-7efa6e20cbe2","Type":"ContainerStarted","Data":"945b9184f9227aa0e84621f7e9e8bbb2ccaf7c7fe681074baa79e748acf15d53"} Jul 28 19:59:55.318219 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.318173 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7mnmq" event={"ID":"96c649c0-feb1-43e1-98d5-c31848ac7b64","Type":"ContainerStarted","Data":"c77fa585dbd1be76bc0a8df60c3bd752abb6590dcd5b1cf8fb2a5b57019829d8"} Jul 28 19:59:55.325316 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.325292 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"deec343b4e5d4277430956b146b8f08ddaf61b98c43e3de5f200511577ec2ba7"} Jul 28 19:59:55.325448 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.325388 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"3f4657f0bde7f2c55d5db75ba74a1cca7671e1f898de6883386064ff4fca8709"} Jul 28 19:59:55.325448 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.325437 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"b185715cddc49972639a817ed800096ff07d255a77bfb1e63203d42728fe08c5"} Jul 28 19:59:55.325557 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.325475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"0373158a86b9bb583c49bb6e10b53a7ae76bfb874a044a4efaa4c3e70770379a"} Jul 28 19:59:55.325557 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.325521 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"c73cd410bd6a87b50f197c921a5a5a727aa8e79d9c3505ae4c21104f2d93763a"} Jul 28 19:59:55.325651 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.325564 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"2caf7ff9f37a0a72c43bfcd4941b7b3f9ebfe449bd6a1f76e920c2f6d656a57b"} Jul 28 19:59:55.326820 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.326772 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z4rd8" podStartSLOduration=11.866070687 podStartE2EDuration="22.326755508s" totalImagesPullingTime="10.460684821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:55.326381739 +0000 UTC m=+22.770692754" watchObservedRunningTime="2026-07-28 19:59:55.326755508 +0000 UTC m=+22.771066526" Jul 28 19:59:55.326989 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.326960 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-255.ec2.internal" podStartSLOduration=21.326951942 podStartE2EDuration="21.326951942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:38.288723797 +0000 UTC m=+5.733034815" watchObservedRunningTime="2026-07-28 19:59:55.326951942 +0000 UTC m=+22.771262958" Jul 28 19:59:55.327058 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.326984 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-knj5g" event={"ID":"fdfc0ce9-10c1-489b-acbf-21e213c48a11","Type":"ContainerStarted","Data":"3a24804145d8122e4ae14025b25bc9946ac453927d0480722ab61d0cd588fa3b"} Jul 28 19:59:55.328719 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.328688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" event={"ID":"099719d8-3d25-418e-b055-a4a1363e2765","Type":"ContainerStarted","Data":"45cba9b4111762f1e4a62eec60b6e5782ac688ef640502d00704c0b50a222d19"} Jul 28 19:59:55.330254 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.330229 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vfv9" event={"ID":"c4e9a8ae-a566-49ae-a090-aae3e55b73e9","Type":"ContainerStarted","Data":"9a3025f0faf921d3dc334882d7d15d45ee6b05a41082c0edc511c33ed04d76dc"} Jul 28 19:59:55.344981 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.344941 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7mnmq" podStartSLOduration=4.034309243 podStartE2EDuration="22.344927884s" totalImagesPullingTime="18.310618641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:55.344739371 +0000 UTC m=+22.789050387" watchObservedRunningTime="2026-07-28 19:59:55.344927884 +0000 UTC m=+22.789238900" Jul 28 19:59:55.362961 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.362921 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v22pp" podStartSLOduration=3.988755683 podStartE2EDuration="22.362909901s" totalImagesPullingTime="18.374154218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:55.362638732 +0000 UTC m=+22.806949750" watchObservedRunningTime="2026-07-28 19:59:55.362909901 +0000 UTC m=+22.807220916" Jul 28 19:59:55.394697 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.394637 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5vfv9" podStartSLOduration=4.086167382 podStartE2EDuration="22.394614986s" totalImagesPullingTime="18.308447604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:55.393966781 +0000 UTC m=+22.838277814" watchObservedRunningTime="2026-07-28 19:59:55.394614986 +0000 UTC m=+22.838926003" Jul 28 19:59:55.408990 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:55.408863 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-knj5g" podStartSLOduration=4.099395352 podStartE2EDuration="22.408845641s" totalImagesPullingTime="18.309450289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:55.40802591 +0000 UTC m=+22.852336927" watchObservedRunningTime="2026-07-28 19:59:55.408845641 +0000 UTC m=+22.853156658" Jul 28 19:59:56.040676 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.040645 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 19:59:56.146095 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.146012 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:56.146273 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:56.146128 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:56.334409 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.334362 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mss6g" event={"ID":"ab6059fd-4193-477e-b5f1-7dc7da431cb2","Type":"ContainerStarted","Data":"14899ad37260a94a014f0cddf4fe7de7c2f8660e77f19d3e2e170d3989db43b8"} Jul 28 19:59:56.336577 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.336546 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" event={"ID":"099719d8-3d25-418e-b055-a4a1363e2765","Type":"ContainerStarted","Data":"1bac19333abc95db6b6dab57380818ea3206b7de8cf64a8bb1732ce42540672e"} Jul 28 19:59:56.348596 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.348553 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mss6g" podStartSLOduration=5.043053517 podStartE2EDuration="23.348536703s" totalImagesPullingTime="18.305483186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:56.348214177 +0000 UTC m=+23.792525193" watchObservedRunningTime="2026-07-28 19:59:56.348536703 +0000 UTC m=+23.792847721" Jul 28 19:59:56.995840 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.995694 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T19:59:56.040669475Z","UUID":"11b6ff41-25d8-49e6-ac93-8955bef2aa6f","Handler":null,"Name":"","Endpoint":""} Jul 28 19:59:56.997672 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.997646 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 19:59:56.997794 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:56.997694 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 19:59:57.146040 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.146014 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:57.146209 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:57.146147 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 19:59:57.146209 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.146190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:57.146389 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:57.146363 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:57.209964 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.209937 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:57.210630 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.210609 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:57.342174 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.342086 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" event={"ID":"099719d8-3d25-418e-b055-a4a1363e2765","Type":"ContainerStarted","Data":"5f7cf7ee9c9bd24eb6a49fd1b2f2dfa41b8d59c3f6d3ffd7d7f4835f4b24cf84"} Jul 28 19:59:57.342471 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.342433 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:57.342980 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.342960 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z4rd8" Jul 28 19:59:57.358944 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:57.358897 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cn4cn" podStartSLOduration=3.795661514 podStartE2EDuration="24.358880387s" totalImagesPullingTime="20.563218873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:59:57.358528501 +0000 UTC m=+24.802839519" watchObservedRunningTime="2026-07-28 19:59:57.358880387 +0000 UTC m=+24.803191402" Jul 28 19:59:58.146486 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:58.146437 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 19:59:58.147054 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:58.146590 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 19:59:58.348786 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:58.348738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"30df165a5dd37c79068439ae42406e16d44a9fd434591063e4c8078318b5728a"} Jul 28 19:59:59.146137 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:59.145887 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 19:59:59.146336 ip-10-0-136-255 kubenswrapper[2581]: I0728 19:59:59.145951 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 19:59:59.146336 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:59.146266 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 19:59:59.146336 ip-10-0-136-255 kubenswrapper[2581]: E0728 19:59:59.146302 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 20:00:00.146732 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:00.146699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:00.147468 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:00.146803 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 20:00:00.354501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:00.354460 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5b556cf-24a3-4fcf-a764-d2f748624105" containerID="6508def356844a79eeeb8108096e726945664a340ef0532087cff4b4d5ba3b2e" exitCode=0 Jul 28 20:00:00.354687 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:00.354502 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerDied","Data":"6508def356844a79eeeb8108096e726945664a340ef0532087cff4b4d5ba3b2e"} Jul 28 20:00:00.358189 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:00.358162 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" event={"ID":"28be3705-4916-4862-bd49-84caadef8558","Type":"ContainerStarted","Data":"ef3273ee328c56e6bac2ff95a545e4e2d044f8845fb7c4235854eefb82ef6f87"} Jul 28 20:00:00.358547 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:00.358532 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 20:00:00.373677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:00.373522 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 20:00:00.439133 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:00.439044 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" podStartSLOduration=9.059073233 podStartE2EDuration="27.439027735s" totalImagesPullingTime="18.379954502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:00.43778653 +0000 UTC m=+27.882097547" watchObservedRunningTime="2026-07-28 20:00:00.439027735 +0000 UTC m=+27.883338752" Jul 28 20:00:01.146561 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:01.146530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:01.146777 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:01.146538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:01.146777 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:01.146661 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 20:00:01.146777 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:01.146706 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 20:00:01.360438 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:01.360401 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 20:00:01.360643 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:01.360478 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 20:00:01.374688 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:01.374662 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 20:00:02.146077 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:02.146048 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:02.146291 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:02.146172 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 20:00:02.364631 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:02.364597 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5b556cf-24a3-4fcf-a764-d2f748624105" containerID="8db7359ed7cfeb32028601e5b1a6b82a570ad528d9623da96ce0397113114a25" exitCode=0 Jul 28 20:00:02.364964 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:02.364676 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerDied","Data":"8db7359ed7cfeb32028601e5b1a6b82a570ad528d9623da96ce0397113114a25"} Jul 28 20:00:03.146452 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:03.146413 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:03.146597 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:03.146530 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 20:00:03.146597 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:03.146580 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:03.146683 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:03.146666 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 20:00:04.146163 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:04.146127 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:04.146577 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:04.146231 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 20:00:05.146358 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:05.146305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:05.146766 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:05.146450 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 20:00:05.146766 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:05.146504 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:05.146766 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:05.146615 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 20:00:05.373285 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:05.373250 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5b556cf-24a3-4fcf-a764-d2f748624105" containerID="324f3b53c8f5d1a5c591d25ba0c2864389333d4347fa15b92a36314cfa68a606" exitCode=0 Jul 28 20:00:05.373503 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:05.373309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerDied","Data":"324f3b53c8f5d1a5c591d25ba0c2864389333d4347fa15b92a36314cfa68a606"} Jul 28 20:00:06.145932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:06.145881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:06.146115 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.145996 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 20:00:06.696988 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:06.696953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:06.697655 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.697128 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 20:00:06.697655 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.697211 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs podName:6bca7392-4c0d-4e95-9e7c-3eb68914d111 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:38.697188361 +0000 UTC m=+66.141499369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs") pod "network-metrics-daemon-xzqcj" (UID: "6bca7392-4c0d-4e95-9e7c-3eb68914d111") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 20:00:06.798365 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:06.798308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:06.798545 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.798439 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 20:00:06.798545 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.798511 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret podName:8f668389-c6ba-44b7-aded-ab82eb3cc122 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:38.798495469 +0000 UTC m=+66.242806468 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret") pod "global-pull-secret-syncer-sq8gz" (UID: "8f668389-c6ba-44b7-aded-ab82eb3cc122") : object "kube-system"/"original-pull-secret" not registered Jul 28 20:00:06.899799 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:06.899728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:06.899954 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.899899 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 20:00:06.899954 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.899924 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 20:00:06.899954 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.899934 2581 projected.go:196] Error preparing data for projected volume kube-api-access-nwlmf for pod openshift-network-diagnostics/network-check-target-vk459: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 20:00:06.900125 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:06.899987 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf podName:f60c1e92-7837-41b8-9bf1-0bebd43c48c7 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:38.899972074 +0000 UTC m=+66.344283073 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nwlmf" (UniqueName: "kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf") pod "network-check-target-vk459" (UID: "f60c1e92-7837-41b8-9bf1-0bebd43c48c7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 20:00:07.146389 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:07.146356 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:07.146574 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:07.146359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:07.146574 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:07.146494 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 20:00:07.146695 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:07.146579 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 20:00:08.146204 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:08.146174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:08.146680 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:08.146285 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 20:00:08.793989 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:08.793954 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sq8gz"] Jul 28 20:00:08.794226 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:08.794131 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:08.794300 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:08.794242 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 20:00:08.798120 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:08.798097 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzqcj"] Jul 28 20:00:08.798232 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:08.798213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:08.798369 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:08.798344 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 20:00:08.798807 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:08.798781 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vk459"] Jul 28 20:00:08.798964 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:08.798904 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:08.799053 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:08.798989 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 20:00:10.388812 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:10.388622 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5b556cf-24a3-4fcf-a764-d2f748624105" containerID="fddd38aff5c0f4fb93c51e646dfd26ca29a99368ec1fcbbea2735a222fee2c89" exitCode=0 Jul 28 20:00:10.388812 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:10.388703 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerDied","Data":"fddd38aff5c0f4fb93c51e646dfd26ca29a99368ec1fcbbea2735a222fee2c89"} Jul 28 20:00:11.146619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:11.146535 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:11.146619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:11.146573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:11.146801 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:11.146545 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:11.146801 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:11.146658 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzqcj" podUID="6bca7392-4c0d-4e95-9e7c-3eb68914d111" Jul 28 20:00:11.146801 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:11.146757 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sq8gz" podUID="8f668389-c6ba-44b7-aded-ab82eb3cc122" Jul 28 20:00:11.146908 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:11.146839 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vk459" podUID="f60c1e92-7837-41b8-9bf1-0bebd43c48c7" Jul 28 20:00:11.395994 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:11.395965 2581 generic.go:356] "Generic (PLEG): container finished" podID="b5b556cf-24a3-4fcf-a764-d2f748624105" containerID="027ba6ef95646cacc4ef292c06322413d925a820f63dbb6ec8f01dc2fe30621c" exitCode=0 Jul 28 20:00:11.396457 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:11.396015 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerDied","Data":"027ba6ef95646cacc4ef292c06322413d925a820f63dbb6ec8f01dc2fe30621c"} Jul 28 20:00:12.401085 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.400998 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" event={"ID":"b5b556cf-24a3-4fcf-a764-d2f748624105","Type":"ContainerStarted","Data":"bab4da095bcffff4689b9a49b31caff5a91e8d221266649d6740d00da4bd8d12"} Jul 28 20:00:12.417196 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.417170 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-255.ec2.internal" event="NodeReady" Jul 28 20:00:12.417337 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.417302 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 20:00:12.425267 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.425226 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nmjk4" podStartSLOduration=8.844872694 podStartE2EDuration="39.425212618s" totalImagesPullingTime="30.580339924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:12.423771075 +0000 UTC m=+39.868082091" watchObservedRunningTime="2026-07-28 20:00:12.425212618 +0000 UTC m=+39.869523635" Jul 28 20:00:12.463011 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.462974 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rgzjl"] Jul 28 20:00:12.487020 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.486995 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mcxfq"] Jul 28 20:00:12.487172 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.487131 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.489251 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.489226 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 20:00:12.489414 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.489232 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pwmd4\"" Jul 28 20:00:12.489414 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.489232 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 20:00:12.489414 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.489308 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 20:00:12.506771 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.506746 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rgzjl"] Jul 28 20:00:12.506771 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.506774 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mcxfq"] Jul 28 20:00:12.506900 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.506865 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.509058 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.509030 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 20:00:12.509202 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.509189 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 20:00:12.509282 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.509266 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 20:00:12.509404 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.509383 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mrvdk\"" Jul 28 20:00:12.509506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.509448 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 20:00:12.568776 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.568747 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-29jh7"] Jul 28 20:00:12.589172 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.589146 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-29jh7"] Jul 28 20:00:12.589312 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.589275 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.591350 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.591306 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 20:00:12.591482 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.591376 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ph26b\"" Jul 28 20:00:12.591482 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.591306 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 20:00:12.641842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.641809 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/179eb78a-da87-466d-b86b-70002c68970a-cert\") pod \"ingress-canary-rgzjl\" (UID: \"179eb78a-da87-466d-b86b-70002c68970a\") " pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.641842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.641843 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhvx\" (UniqueName: \"kubernetes.io/projected/179eb78a-da87-466d-b86b-70002c68970a-kube-api-access-wxhvx\") pod \"ingress-canary-rgzjl\" (UID: \"179eb78a-da87-466d-b86b-70002c68970a\") " pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.642056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.641861 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d99636cc-df6d-421b-96bf-1523e71f523f-crio-socket\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.642056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.641894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clj7b\" (UniqueName: \"kubernetes.io/projected/d99636cc-df6d-421b-96bf-1523e71f523f-kube-api-access-clj7b\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.642056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.641965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d99636cc-df6d-421b-96bf-1523e71f523f-data-volume\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.642056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.642004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d99636cc-df6d-421b-96bf-1523e71f523f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.642056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.642052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d99636cc-df6d-421b-96bf-1523e71f523f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743141 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743111 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/179eb78a-da87-466d-b86b-70002c68970a-cert\") pod \"ingress-canary-rgzjl\" (UID: \"179eb78a-da87-466d-b86b-70002c68970a\") " pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.743309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhvx\" (UniqueName: \"kubernetes.io/projected/179eb78a-da87-466d-b86b-70002c68970a-kube-api-access-wxhvx\") pod \"ingress-canary-rgzjl\" (UID: \"179eb78a-da87-466d-b86b-70002c68970a\") " pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.743309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d99636cc-df6d-421b-96bf-1523e71f523f-crio-socket\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743217 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clj7b\" (UniqueName: \"kubernetes.io/projected/d99636cc-df6d-421b-96bf-1523e71f523f-kube-api-access-clj7b\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d99636cc-df6d-421b-96bf-1523e71f523f-data-volume\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d99636cc-df6d-421b-96bf-1523e71f523f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d99636cc-df6d-421b-96bf-1523e71f523f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17856a56-d18c-43ad-9049-0aad056adbd8-config-volume\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.743567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17856a56-d18c-43ad-9049-0aad056adbd8-tmp-dir\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.743567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743496 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d99636cc-df6d-421b-96bf-1523e71f523f-crio-socket\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17856a56-d18c-43ad-9049-0aad056adbd8-metrics-tls\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.743567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmkd\" (UniqueName: \"kubernetes.io/projected/17856a56-d18c-43ad-9049-0aad056adbd8-kube-api-access-qgmkd\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.743808 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743626 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d99636cc-df6d-421b-96bf-1523e71f523f-data-volume\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.743919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.743900 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d99636cc-df6d-421b-96bf-1523e71f523f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.747163 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.747138 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d99636cc-df6d-421b-96bf-1523e71f523f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.747274 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.747252 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/179eb78a-da87-466d-b86b-70002c68970a-cert\") pod \"ingress-canary-rgzjl\" (UID: \"179eb78a-da87-466d-b86b-70002c68970a\") " pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.750621 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.750595 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clj7b\" (UniqueName: \"kubernetes.io/projected/d99636cc-df6d-421b-96bf-1523e71f523f-kube-api-access-clj7b\") pod \"insights-runtime-extractor-mcxfq\" (UID: \"d99636cc-df6d-421b-96bf-1523e71f523f\") " pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.751036 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.751008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhvx\" (UniqueName: \"kubernetes.io/projected/179eb78a-da87-466d-b86b-70002c68970a-kube-api-access-wxhvx\") pod \"ingress-canary-rgzjl\" (UID: \"179eb78a-da87-466d-b86b-70002c68970a\") " pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.795887 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.795857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rgzjl" Jul 28 20:00:12.816059 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.816037 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mcxfq" Jul 28 20:00:12.844424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.844396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17856a56-d18c-43ad-9049-0aad056adbd8-metrics-tls\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.844528 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.844430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmkd\" (UniqueName: \"kubernetes.io/projected/17856a56-d18c-43ad-9049-0aad056adbd8-kube-api-access-qgmkd\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.844643 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.844618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17856a56-d18c-43ad-9049-0aad056adbd8-config-volume\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.844686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.844658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17856a56-d18c-43ad-9049-0aad056adbd8-tmp-dir\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.844961 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.844937 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17856a56-d18c-43ad-9049-0aad056adbd8-tmp-dir\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.845170 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.845148 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17856a56-d18c-43ad-9049-0aad056adbd8-config-volume\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.846720 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.846699 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17856a56-d18c-43ad-9049-0aad056adbd8-metrics-tls\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.852682 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.852656 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmkd\" (UniqueName: \"kubernetes.io/projected/17856a56-d18c-43ad-9049-0aad056adbd8-kube-api-access-qgmkd\") pod \"dns-default-29jh7\" (UID: \"17856a56-d18c-43ad-9049-0aad056adbd8\") " pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:12.899967 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:12.899933 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:13.002123 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.002059 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rgzjl"] Jul 28 20:00:13.006262 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:13.006234 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179eb78a_da87_466d_b86b_70002c68970a.slice/crio-902ebdff795f70ef6eef0afa90190976282a33a4f82c6c8ecee8d4ea1ee804fb WatchSource:0}: Error finding container 902ebdff795f70ef6eef0afa90190976282a33a4f82c6c8ecee8d4ea1ee804fb: Status 404 returned error can't find the container with id 902ebdff795f70ef6eef0afa90190976282a33a4f82c6c8ecee8d4ea1ee804fb Jul 28 20:00:13.025845 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.025819 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mcxfq"] Jul 28 20:00:13.028879 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:13.028852 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99636cc_df6d_421b_96bf_1523e71f523f.slice/crio-77576911481ab9d01cb2bf8d8298eae7798829d339f804498956050e3eaab574 WatchSource:0}: Error finding container 77576911481ab9d01cb2bf8d8298eae7798829d339f804498956050e3eaab574: Status 404 returned error can't find the container with id 77576911481ab9d01cb2bf8d8298eae7798829d339f804498956050e3eaab574 Jul 28 20:00:13.069203 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.069166 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-29jh7"] Jul 28 20:00:13.073126 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:13.073099 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17856a56_d18c_43ad_9049_0aad056adbd8.slice/crio-d69182c38dfb74fba147fff4ceb0772f81e1198d288c314cd6394c57257c5281 WatchSource:0}: Error finding container d69182c38dfb74fba147fff4ceb0772f81e1198d288c314cd6394c57257c5281: Status 404 returned error can't find the container with id d69182c38dfb74fba147fff4ceb0772f81e1198d288c314cd6394c57257c5281 Jul 28 20:00:13.147154 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.147117 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:13.147154 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.147152 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:13.147407 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.147367 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:13.149663 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.149637 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 20:00:13.149663 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.149637 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 20:00:13.149946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.149640 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kg9b\"" Jul 28 20:00:13.149946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.149695 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 20:00:13.149946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.149717 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 20:00:13.150074 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.150003 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gjwl7\"" Jul 28 20:00:13.404242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.404145 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29jh7" event={"ID":"17856a56-d18c-43ad-9049-0aad056adbd8","Type":"ContainerStarted","Data":"d69182c38dfb74fba147fff4ceb0772f81e1198d288c314cd6394c57257c5281"} Jul 28 20:00:13.405227 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.405205 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcxfq" event={"ID":"d99636cc-df6d-421b-96bf-1523e71f523f","Type":"ContainerStarted","Data":"77576911481ab9d01cb2bf8d8298eae7798829d339f804498956050e3eaab574"} Jul 28 20:00:13.406194 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:13.406171 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rgzjl" event={"ID":"179eb78a-da87-466d-b86b-70002c68970a","Type":"ContainerStarted","Data":"902ebdff795f70ef6eef0afa90190976282a33a4f82c6c8ecee8d4ea1ee804fb"} Jul 28 20:00:15.452884 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.452849 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9"] Jul 28 20:00:15.468835 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.468803 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9"] Jul 28 20:00:15.469009 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.468903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" Jul 28 20:00:15.470992 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.470965 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 20:00:15.470992 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.470991 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vd9gp\"" Jul 28 20:00:15.666849 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.666816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2v8l9\" (UID: \"a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" Jul 28 20:00:15.768182 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.768144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2v8l9\" (UID: \"a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" Jul 28 20:00:15.770934 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.770909 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2v8l9\" (UID: \"a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" Jul 28 20:00:15.780678 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:15.780649 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" Jul 28 20:00:20.170967 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.170889 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9"] Jul 28 20:00:20.174306 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:20.174278 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bcc05b_3fc3_4ae0_97a0_b382cb647ecb.slice/crio-2182d08698216c815de6f446cb9eb0c4c09a091a0858bef7b47eacf9af7c7bc3 WatchSource:0}: Error finding container 2182d08698216c815de6f446cb9eb0c4c09a091a0858bef7b47eacf9af7c7bc3: Status 404 returned error can't find the container with id 2182d08698216c815de6f446cb9eb0c4c09a091a0858bef7b47eacf9af7c7bc3 Jul 28 20:00:20.427645 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.427611 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rgzjl" event={"ID":"179eb78a-da87-466d-b86b-70002c68970a","Type":"ContainerStarted","Data":"f0ad39169b83155c561a6d59c1499cbe171d71131046782d2a02105520c48f0a"} Jul 28 20:00:20.429200 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.429124 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" event={"ID":"a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb","Type":"ContainerStarted","Data":"2182d08698216c815de6f446cb9eb0c4c09a091a0858bef7b47eacf9af7c7bc3"} Jul 28 20:00:20.431751 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.431728 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29jh7" event={"ID":"17856a56-d18c-43ad-9049-0aad056adbd8","Type":"ContainerStarted","Data":"334fda993f0f262ed76f8d8b0ef23335de8f1cbda8cc60c9e7fc02969ab9bd7f"} Jul 28 20:00:20.442706 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.442646 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rgzjl" podStartSLOduration=1.4106097 podStartE2EDuration="8.442627651s" totalImagesPullingTime="7.032017951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:20.442360986 +0000 UTC m=+47.886672003" watchObservedRunningTime="2026-07-28 20:00:20.442627651 +0000 UTC m=+47.886938671" Jul 28 20:00:20.511711 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.511682 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86dc84b9d7-7vzzf"] Jul 28 20:00:20.528152 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.528128 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dc84b9d7-7vzzf"] Jul 28 20:00:20.528289 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.528213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.530557 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.530529 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 20:00:20.530689 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.530565 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 20:00:20.530689 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.530578 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 20:00:20.530689 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.530632 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 20:00:20.530689 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.530539 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 20:00:20.530908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.530741 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 20:00:20.530999 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.530981 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hmrl6\"" Jul 28 20:00:20.531174 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.531158 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 20:00:20.598489 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.598399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-config\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.598489 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.598459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-serving-cert\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.598489 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.598480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rsr\" (UniqueName: \"kubernetes.io/projected/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-kube-api-access-52rsr\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.598715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.598527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-service-ca\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.598715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.598557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-oauth-serving-cert\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.598715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.598582 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-oauth-config\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.699445 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.699406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-config\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.699641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.699477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-serving-cert\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.699641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.699499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rsr\" (UniqueName: \"kubernetes.io/projected/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-kube-api-access-52rsr\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.699641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.699533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-service-ca\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.699641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.699558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-oauth-serving-cert\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.699641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.699594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-oauth-config\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.700100 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.700070 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-config\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.700201 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.700133 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-oauth-serving-cert\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.700301 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.700277 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-service-ca\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.702056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.702028 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-oauth-config\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.702151 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.702070 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-serving-cert\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.707035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.707015 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rsr\" (UniqueName: \"kubernetes.io/projected/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-kube-api-access-52rsr\") pod \"console-86dc84b9d7-7vzzf\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:20.838080 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:20.838045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:21.005486 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:21.005451 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dc84b9d7-7vzzf"] Jul 28 20:00:21.010826 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:21.010798 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcb0a3a_c07c_459e_ab58_3c31ca47062d.slice/crio-f8ee3b51724ba4f3810c4a4e8d4ef1be8c9703388d14144fcd8a20e47a2d708b WatchSource:0}: Error finding container f8ee3b51724ba4f3810c4a4e8d4ef1be8c9703388d14144fcd8a20e47a2d708b: Status 404 returned error can't find the container with id f8ee3b51724ba4f3810c4a4e8d4ef1be8c9703388d14144fcd8a20e47a2d708b Jul 28 20:00:21.437447 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:21.437356 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc84b9d7-7vzzf" event={"ID":"2dcb0a3a-c07c-459e-ab58-3c31ca47062d","Type":"ContainerStarted","Data":"f8ee3b51724ba4f3810c4a4e8d4ef1be8c9703388d14144fcd8a20e47a2d708b"} Jul 28 20:00:21.439831 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:21.439801 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-29jh7" event={"ID":"17856a56-d18c-43ad-9049-0aad056adbd8","Type":"ContainerStarted","Data":"103ddfd4287522fe0e935e0fee2712deeb75cdfc414f65226023cab8a86580a6"} Jul 28 20:00:21.440315 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:21.440270 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:21.442213 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:21.442140 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcxfq" event={"ID":"d99636cc-df6d-421b-96bf-1523e71f523f","Type":"ContainerStarted","Data":"4b74d6eefae3c58adc6f6e85af53e57508087648bdad6b39441f5f80d4a15874"} Jul 28 20:00:21.455116 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:21.455063 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-29jh7" podStartSLOduration=2.489407349 podStartE2EDuration="9.455046211s" totalImagesPullingTime="6.965638862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:21.45461998 +0000 UTC m=+48.898930997" watchObservedRunningTime="2026-07-28 20:00:21.455046211 +0000 UTC m=+48.899357228" Jul 28 20:00:25.464449 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:25.464409 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcxfq" event={"ID":"d99636cc-df6d-421b-96bf-1523e71f523f","Type":"ContainerStarted","Data":"4ed02b297a3435099d628331c24c1bbd92165ae663930ce289ba06b31252f9f4"} Jul 28 20:00:25.465993 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:25.465963 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" event={"ID":"a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb","Type":"ContainerStarted","Data":"028272ee5cd6686ace53b713b1adfc7c057cad561137601538783a2cbb64a77f"} Jul 28 20:00:25.466183 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:25.466163 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" Jul 28 20:00:25.471529 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:25.471504 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" Jul 28 20:00:25.481107 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:25.481057 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2v8l9" podStartSLOduration=6.098371257 podStartE2EDuration="10.48103943s" totalImagesPullingTime="4.382668173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:25.47977898 +0000 UTC m=+52.924089995" watchObservedRunningTime="2026-07-28 20:00:25.48103943 +0000 UTC m=+52.925350447" Jul 28 20:00:26.528646 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.528603 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zvs4w"] Jul 28 20:00:26.544413 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.544389 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zvs4w"] Jul 28 20:00:26.544584 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.544481 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.547372 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.547346 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 20:00:26.547372 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.547355 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 20:00:26.547562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.547346 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 20:00:26.547562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.547379 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 20:00:26.547562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.547414 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 20:00:26.547562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.547355 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z4w24\"" Jul 28 20:00:26.644822 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.644784 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t826q\" (UniqueName: \"kubernetes.io/projected/3cf84def-1c6b-4cd9-8d66-8798b4360452-kube-api-access-t826q\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.645015 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.644894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf84def-1c6b-4cd9-8d66-8798b4360452-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.645015 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.644927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf84def-1c6b-4cd9-8d66-8798b4360452-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.645015 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.644960 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cf84def-1c6b-4cd9-8d66-8798b4360452-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.745857 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.745818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t826q\" (UniqueName: \"kubernetes.io/projected/3cf84def-1c6b-4cd9-8d66-8798b4360452-kube-api-access-t826q\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.746025 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.745865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf84def-1c6b-4cd9-8d66-8798b4360452-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.746025 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.745954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf84def-1c6b-4cd9-8d66-8798b4360452-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.746139 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.746073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cf84def-1c6b-4cd9-8d66-8798b4360452-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.746560 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.746531 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf84def-1c6b-4cd9-8d66-8798b4360452-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.748312 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.748288 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf84def-1c6b-4cd9-8d66-8798b4360452-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.748438 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.748381 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cf84def-1c6b-4cd9-8d66-8798b4360452-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.753635 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.753615 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t826q\" (UniqueName: \"kubernetes.io/projected/3cf84def-1c6b-4cd9-8d66-8798b4360452-kube-api-access-t826q\") pod \"prometheus-operator-84d69f794d-zvs4w\" (UID: \"3cf84def-1c6b-4cd9-8d66-8798b4360452\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:26.853042 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:26.853016 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" Jul 28 20:00:27.015030 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:27.014999 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zvs4w"] Jul 28 20:00:27.019541 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:27.019502 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf84def_1c6b_4cd9_8d66_8798b4360452.slice/crio-173b99b7636e85d9c564d92a3f5af00dcc0c3e378d5046261be97a977f2f99f3 WatchSource:0}: Error finding container 173b99b7636e85d9c564d92a3f5af00dcc0c3e378d5046261be97a977f2f99f3: Status 404 returned error can't find the container with id 173b99b7636e85d9c564d92a3f5af00dcc0c3e378d5046261be97a977f2f99f3 Jul 28 20:00:27.473831 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:27.473791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc84b9d7-7vzzf" event={"ID":"2dcb0a3a-c07c-459e-ab58-3c31ca47062d","Type":"ContainerStarted","Data":"7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc"} Jul 28 20:00:27.475229 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:27.475165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" event={"ID":"3cf84def-1c6b-4cd9-8d66-8798b4360452","Type":"ContainerStarted","Data":"173b99b7636e85d9c564d92a3f5af00dcc0c3e378d5046261be97a977f2f99f3"} Jul 28 20:00:27.494993 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:27.494951 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86dc84b9d7-7vzzf" podStartSLOduration=1.6619152320000001 podStartE2EDuration="7.494935774s" totalImagesPullingTime="5.833020542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:27.493989925 +0000 UTC m=+54.938300943" watchObservedRunningTime="2026-07-28 20:00:27.494935774 +0000 UTC m=+54.939246790" Jul 28 20:00:30.490051 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.490017 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mcxfq" event={"ID":"d99636cc-df6d-421b-96bf-1523e71f523f","Type":"ContainerStarted","Data":"8936ac6441cc01cbb2e73a10c1a1ae3eabc80a1e985bcefcf11852da25123aca"} Jul 28 20:00:30.491819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.491792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" event={"ID":"3cf84def-1c6b-4cd9-8d66-8798b4360452","Type":"ContainerStarted","Data":"c7da51d8c583da9860dcc01284edaefb033a25ed36d17ebdd7838d0b6daf8771"} Jul 28 20:00:30.491919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.491830 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" event={"ID":"3cf84def-1c6b-4cd9-8d66-8798b4360452","Type":"ContainerStarted","Data":"856dc4bde34610384a64837e59a710a34e52c97ce0c1c5b1e5f3cd1d34b7ddb8"} Jul 28 20:00:30.507883 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.507838 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mcxfq" podStartSLOduration=1.648630176 podStartE2EDuration="18.507825004s" totalImagesPullingTime="16.859194828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:00:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:30.506972207 +0000 UTC m=+57.951283234" watchObservedRunningTime="2026-07-28 20:00:30.507825004 +0000 UTC m=+57.952136020" Jul 28 20:00:30.527566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.527503 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-zvs4w" podStartSLOduration=1.421716234 podStartE2EDuration="4.527481585s" totalImagesPullingTime="3.105765351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:30.526491697 +0000 UTC m=+57.970802715" watchObservedRunningTime="2026-07-28 20:00:30.527481585 +0000 UTC m=+57.971792603" Jul 28 20:00:30.839275 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.839173 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:30.839275 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.839261 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:30.843820 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:30.843800 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:31.447201 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.447172 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-29jh7" Jul 28 20:00:31.493615 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.493590 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7559966c56-zdbb4"] Jul 28 20:00:31.496121 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.496096 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.499016 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.498989 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:00:31.502672 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.502649 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 20:00:31.509451 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.509430 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7559966c56-zdbb4"] Jul 28 20:00:31.582755 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.582723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-service-ca\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.582942 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.582784 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-trusted-ca-bundle\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.582942 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.582854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-config\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.583048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.582939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-oauth-serving-cert\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.587808 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.587772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-serving-cert\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.587938 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.587852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-oauth-config\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.587992 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.587936 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmlr\" (UniqueName: \"kubernetes.io/projected/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-kube-api-access-qlmlr\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689095 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-trusted-ca-bundle\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689095 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-config\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689369 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689241 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-oauth-serving-cert\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689369 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689302 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-serving-cert\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-oauth-config\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmlr\" (UniqueName: \"kubernetes.io/projected/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-kube-api-access-qlmlr\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-service-ca\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.689941 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689913 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-config\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.690008 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.689990 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-oauth-serving-cert\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.690059 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.690040 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-service-ca\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.690300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.690278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-trusted-ca-bundle\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.691767 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.691747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-oauth-config\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.691872 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.691849 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-serving-cert\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.697077 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.697058 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmlr\" (UniqueName: \"kubernetes.io/projected/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-kube-api-access-qlmlr\") pod \"console-7559966c56-zdbb4\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.806268 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.806235 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:31.953118 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:31.953087 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7559966c56-zdbb4"] Jul 28 20:00:31.955883 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:31.955861 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4695943_4ed7_4890_b92a_fa1fc0ce11ca.slice/crio-fe02f19bc6e819e344cc3d0362d62abbaf06924765b3fdf507370d98ba453bf3 WatchSource:0}: Error finding container fe02f19bc6e819e344cc3d0362d62abbaf06924765b3fdf507370d98ba453bf3: Status 404 returned error can't find the container with id fe02f19bc6e819e344cc3d0362d62abbaf06924765b3fdf507370d98ba453bf3 Jul 28 20:00:32.499510 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.499472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7559966c56-zdbb4" event={"ID":"d4695943-4ed7-4890-b92a-fa1fc0ce11ca","Type":"ContainerStarted","Data":"f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308"} Jul 28 20:00:32.499882 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.499527 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7559966c56-zdbb4" event={"ID":"d4695943-4ed7-4890-b92a-fa1fc0ce11ca","Type":"ContainerStarted","Data":"fe02f19bc6e819e344cc3d0362d62abbaf06924765b3fdf507370d98ba453bf3"} Jul 28 20:00:32.516887 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.516839 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7559966c56-zdbb4" podStartSLOduration=1.516824073 podStartE2EDuration="1.516824073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:32.515372985 +0000 UTC m=+59.959683997" watchObservedRunningTime="2026-07-28 20:00:32.516824073 +0000 UTC m=+59.961135089" Jul 28 20:00:32.915102 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.915018 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb"] Jul 28 20:00:32.917507 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.917482 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:32.919386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.919362 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 20:00:32.919512 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.919493 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 20:00:32.919586 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.919522 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5c82g\"" Jul 28 20:00:32.928880 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.928856 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb"] Jul 28 20:00:32.934405 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.934384 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d"] Jul 28 20:00:32.936863 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.936848 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:32.938824 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.938800 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 20:00:32.938920 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.938828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 20:00:32.938920 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.938868 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 20:00:32.938920 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.938887 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qm8d9\"" Jul 28 20:00:32.953044 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.952794 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d"] Jul 28 20:00:32.953370 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.953288 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c8kjr"] Jul 28 20:00:32.956095 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.956075 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:32.958036 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.958018 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 20:00:32.958193 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.958048 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 20:00:32.958264 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.958094 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q6dkm\"" Jul 28 20:00:32.958264 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:32.958182 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 20:00:33.000247 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000344 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.000446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000390 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-root\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.000609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-metrics-client-ca\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-wtmp\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-sys\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000584 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.000798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8d45\" (UniqueName: \"kubernetes.io/projected/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-kube-api-access-r8d45\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.000798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-tls\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000685 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.000798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.000798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbdwp\" (UniqueName: \"kubernetes.io/projected/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-kube-api-access-fbdwp\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000829 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphst\" (UniqueName: \"kubernetes.io/projected/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-api-access-nphst\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.000996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000865 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-textfile\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.000996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.000996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000924 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.000996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.000939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.101233 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-sys\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.101233 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8d45\" (UniqueName: \"kubernetes.io/projected/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-kube-api-access-r8d45\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-tls\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101309 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-sys\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101348 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbdwp\" (UniqueName: \"kubernetes.io/projected/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-kube-api-access-fbdwp\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101382 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphst\" (UniqueName: \"kubernetes.io/projected/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-api-access-nphst\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-textfile\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.101501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101578 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-root\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-metrics-client-ca\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-wtmp\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101752 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101860 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-wtmp\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.101908 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-root\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.102047 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.102064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.102059 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.102630 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.102606 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.102904 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.102878 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.103979 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.103947 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-textfile\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.104607 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.104552 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-tls\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.104704 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.104629 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-metrics-client-ca\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.104939 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.104907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.105018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.104996 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.105118 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.105102 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.105332 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.105296 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.105735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.105719 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.113867 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.113835 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphst\" (UniqueName: \"kubernetes.io/projected/591c6c2d-05f6-43a0-a8b9-4fbc05852c2f-kube-api-access-nphst\") pod \"kube-state-metrics-56d9f944b4-v9k6d\" (UID: \"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.114216 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.114197 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8d45\" (UniqueName: \"kubernetes.io/projected/f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6-kube-api-access-r8d45\") pod \"openshift-state-metrics-b8f49f5c6-5qckb\" (UID: \"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.114672 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.114651 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbdwp\" (UniqueName: \"kubernetes.io/projected/2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0-kube-api-access-fbdwp\") pod \"node-exporter-c8kjr\" (UID: \"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0\") " pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.227389 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.227297 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" Jul 28 20:00:33.245499 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.245464 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" Jul 28 20:00:33.268847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.268821 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c8kjr" Jul 28 20:00:33.340119 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:33.340086 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2541d2b1_8fd7_4d4c_9dfa_e73f4b61f9a0.slice/crio-b00e43bd75e5046d84f0a7bd48f8e0abeae21cd4b750ee8600644a023a2dc45c WatchSource:0}: Error finding container b00e43bd75e5046d84f0a7bd48f8e0abeae21cd4b750ee8600644a023a2dc45c: Status 404 returned error can't find the container with id b00e43bd75e5046d84f0a7bd48f8e0abeae21cd4b750ee8600644a023a2dc45c Jul 28 20:00:33.388477 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.388441 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kk9q6" Jul 28 20:00:33.398508 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.398478 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb"] Jul 28 20:00:33.402276 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:33.402239 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d3ebc7_9793_4df3_bfdf_1598eb7ee9d6.slice/crio-9ae3c3155a017a6b6d679a931c8a3d681e0d99067fefb36bceee0cf09cd93dd6 WatchSource:0}: Error finding container 9ae3c3155a017a6b6d679a931c8a3d681e0d99067fefb36bceee0cf09cd93dd6: Status 404 returned error can't find the container with id 9ae3c3155a017a6b6d679a931c8a3d681e0d99067fefb36bceee0cf09cd93dd6 Jul 28 20:00:33.431243 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.431159 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d"] Jul 28 20:00:33.435134 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:33.435112 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591c6c2d_05f6_43a0_a8b9_4fbc05852c2f.slice/crio-9c44cc6e10050b5827ec483eadc29865f80ce1f41979a73bf76d9cb3ce3c9a77 WatchSource:0}: Error finding container 9c44cc6e10050b5827ec483eadc29865f80ce1f41979a73bf76d9cb3ce3c9a77: Status 404 returned error can't find the container with id 9c44cc6e10050b5827ec483eadc29865f80ce1f41979a73bf76d9cb3ce3c9a77 Jul 28 20:00:33.504128 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.504091 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" event={"ID":"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6","Type":"ContainerStarted","Data":"9ae3c3155a017a6b6d679a931c8a3d681e0d99067fefb36bceee0cf09cd93dd6"} Jul 28 20:00:33.505475 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.505447 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8kjr" event={"ID":"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0","Type":"ContainerStarted","Data":"b00e43bd75e5046d84f0a7bd48f8e0abeae21cd4b750ee8600644a023a2dc45c"} Jul 28 20:00:33.506531 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.506506 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" event={"ID":"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f","Type":"ContainerStarted","Data":"9c44cc6e10050b5827ec483eadc29865f80ce1f41979a73bf76d9cb3ce3c9a77"} Jul 28 20:00:33.972764 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:33.972690 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:00:34.008399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.008368 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:00:34.008565 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.008464 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.012044 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012020 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 20:00:34.012186 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012053 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 20:00:34.012186 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012077 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 20:00:34.012186 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012077 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 20:00:34.012429 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012405 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 20:00:34.012597 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012579 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 20:00:34.012667 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012648 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 20:00:34.012898 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.012881 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-52z6n\"" Jul 28 20:00:34.013116 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.013102 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 20:00:34.013163 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.013130 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 20:00:34.109957 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.109922 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.109957 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.109958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.109985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-web-config\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-config-volume\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-config-out\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrmq\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-kube-api-access-kdrmq\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110296 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.110453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.110376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.210841 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.210805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrmq\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-kube-api-access-kdrmq\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.210871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.210896 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.210927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211215 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211267 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211347 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211275 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211347 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-web-config\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211347 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-config-volume\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211372 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-config-out\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.211681 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211523 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.212052 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.211933 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.214060 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.214034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.214169 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:34.214148 2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 28 20:00:34.214250 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.214212 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-config-out\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.214250 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:34.214231 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls podName:62482197-a74a-4d25-8490-36f4559ec673 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:34.714212028 +0000 UTC m=+62.158523028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "62482197-a74a-4d25-8490-36f4559ec673") : secret "alertmanager-main-tls" not found Jul 28 20:00:34.214429 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:34.214403 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle podName:62482197-a74a-4d25-8490-36f4559ec673 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:34.714386742 +0000 UTC m=+62.158697752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "62482197-a74a-4d25-8490-36f4559ec673") : configmap references non-existent config key: ca-bundle.crt Jul 28 20:00:34.214641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.214618 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-config-volume\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.214953 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.214896 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-tls-assets\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.215426 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.215404 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.215426 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.215418 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.215763 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.215730 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.216410 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.216385 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-web-config\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.220067 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.220036 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrmq\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-kube-api-access-kdrmq\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.511274 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.511235 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" event={"ID":"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6","Type":"ContainerStarted","Data":"6a65f401a2863dd6833fbb548ae48c15fa096a34c4d92824c9fa3edb57791ce8"} Jul 28 20:00:34.511751 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.511299 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" event={"ID":"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6","Type":"ContainerStarted","Data":"1959e81655340960c659c696eb9818e68ce448d28cbc3d9d4b8379cdf8116924"} Jul 28 20:00:34.717577 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.717536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.717784 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.717662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.718803 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.718751 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.721140 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.720986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:34.871263 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.871184 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-54bbc586dd-qj699"] Jul 28 20:00:34.910439 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.910396 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54bbc586dd-qj699"] Jul 28 20:00:34.910601 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.910577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:34.912774 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.912700 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 20:00:34.912906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.912806 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4o7p8967ig1a9\"" Jul 28 20:00:34.912906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.912823 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-54f62\"" Jul 28 20:00:34.912906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.912837 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 20:00:34.913112 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.913096 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 20:00:34.913112 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.913106 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 20:00:34.913228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.913152 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 20:00:34.918491 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:34.918412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:00:35.020048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020010 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-metrics-client-ca\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.020221 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020100 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.020221 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.020348 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020232 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.020348 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020301 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-grpc-tls\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.020454 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020347 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.020454 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-tls\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.020454 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.020390 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzh6k\" (UniqueName: \"kubernetes.io/projected/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-kube-api-access-jzh6k\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121426 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121341 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-grpc-tls\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121580 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121580 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-tls\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121580 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121563 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzh6k\" (UniqueName: \"kubernetes.io/projected/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-kube-api-access-jzh6k\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121737 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-metrics-client-ca\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121737 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121861 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.121861 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.121805 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.122370 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.122342 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-metrics-client-ca\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.124476 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.124444 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-tls\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.124590 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.124498 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.124590 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.124498 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.124846 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.124825 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.124846 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.124836 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-grpc-tls\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.125066 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.125048 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.130026 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.130007 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzh6k\" (UniqueName: \"kubernetes.io/projected/c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78-kube-api-access-jzh6k\") pod \"thanos-querier-54bbc586dd-qj699\" (UID: \"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78\") " pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:35.221775 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:35.221742 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:36.754271 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:36.754242 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:00:36.758476 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:36.758445 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62482197_a74a_4d25_8490_36f4559ec673.slice/crio-c8c3fdf5295d62f4d41fd1f2121ba8f20250ab939d7e44e3dd8289dab29843e7 WatchSource:0}: Error finding container c8c3fdf5295d62f4d41fd1f2121ba8f20250ab939d7e44e3dd8289dab29843e7: Status 404 returned error can't find the container with id c8c3fdf5295d62f4d41fd1f2121ba8f20250ab939d7e44e3dd8289dab29843e7 Jul 28 20:00:36.774719 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:36.774685 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-54bbc586dd-qj699"] Jul 28 20:00:36.779895 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:36.779861 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d6c98f_8cc3_42e7_a38d_08af6b4a9b78.slice/crio-80ada77bfe09ed0b35c4548dba50ff255ca3da55d46c2c9a4b5a3ead0c7f3314 WatchSource:0}: Error finding container 80ada77bfe09ed0b35c4548dba50ff255ca3da55d46c2c9a4b5a3ead0c7f3314: Status 404 returned error can't find the container with id 80ada77bfe09ed0b35c4548dba50ff255ca3da55d46c2c9a4b5a3ead0c7f3314 Jul 28 20:00:37.177804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.177724 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc"] Jul 28 20:00:37.179953 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.179933 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.182853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.182789 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 20:00:37.182853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.182831 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 20:00:37.182853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.182838 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5htk0mbf1ndda\"" Jul 28 20:00:37.182853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.182848 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fnbgq\"" Jul 28 20:00:37.183137 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.182863 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 20:00:37.183471 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.183454 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 20:00:37.196843 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.196813 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc"] Jul 28 20:00:37.243686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.243652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c62c6f2b-4184-4048-84f1-955575f8a2d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.243686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.243687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-client-ca-bundle\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.243930 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.243771 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-secret-metrics-server-tls\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.243930 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.243838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdc58\" (UniqueName: \"kubernetes.io/projected/c62c6f2b-4184-4048-84f1-955575f8a2d9-kube-api-access-hdc58\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.243930 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.243890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c62c6f2b-4184-4048-84f1-955575f8a2d9-metrics-server-audit-profiles\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.244041 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.243950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-secret-metrics-server-client-certs\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.244041 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.243977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c62c6f2b-4184-4048-84f1-955575f8a2d9-audit-log\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.344962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c62c6f2b-4184-4048-84f1-955575f8a2d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-client-ca-bundle\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345261 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-secret-metrics-server-tls\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345261 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc58\" (UniqueName: \"kubernetes.io/projected/c62c6f2b-4184-4048-84f1-955575f8a2d9-kube-api-access-hdc58\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345261 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c62c6f2b-4184-4048-84f1-955575f8a2d9-metrics-server-audit-profiles\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345261 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-secret-metrics-server-client-certs\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345261 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c62c6f2b-4184-4048-84f1-955575f8a2d9-audit-log\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345650 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345626 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c62c6f2b-4184-4048-84f1-955575f8a2d9-audit-log\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.345919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.345897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c62c6f2b-4184-4048-84f1-955575f8a2d9-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.346258 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.346233 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c62c6f2b-4184-4048-84f1-955575f8a2d9-metrics-server-audit-profiles\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.347565 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.347541 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-client-ca-bundle\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.347641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.347623 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-secret-metrics-server-client-certs\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.347686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.347665 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c62c6f2b-4184-4048-84f1-955575f8a2d9-secret-metrics-server-tls\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.353895 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.353874 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdc58\" (UniqueName: \"kubernetes.io/projected/c62c6f2b-4184-4048-84f1-955575f8a2d9-kube-api-access-hdc58\") pod \"metrics-server-7bc6fdb7cf-v75qc\" (UID: \"c62c6f2b-4184-4048-84f1-955575f8a2d9\") " pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.489192 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.489158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:37.531687 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.531649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" event={"ID":"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f","Type":"ContainerStarted","Data":"a8d070f6bef635ea8900c4b1d08c328c1cbf95fcfa751e027bd7dd9f5b81f70b"} Jul 28 20:00:37.531862 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.531725 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" event={"ID":"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f","Type":"ContainerStarted","Data":"d16535fd60fc9cafb00cd45ff8ca3f2fb5ad82f081021df96d069e32744a476f"} Jul 28 20:00:37.531862 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.531759 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" event={"ID":"591c6c2d-05f6-43a0-a8b9-4fbc05852c2f","Type":"ContainerStarted","Data":"e2cbda44f85ad21b07124db134c4f1dc58dd57ea1ebe083b748daa1c9b8c64e2"} Jul 28 20:00:37.534013 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.533985 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" event={"ID":"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78","Type":"ContainerStarted","Data":"80ada77bfe09ed0b35c4548dba50ff255ca3da55d46c2c9a4b5a3ead0c7f3314"} Jul 28 20:00:37.541125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.538586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" event={"ID":"f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6","Type":"ContainerStarted","Data":"43349d9cc05e8d07679195f893a39adbaa741f2c177a620ecd233c4829cb0f2a"} Jul 28 20:00:37.541125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.540604 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerStarted","Data":"c8c3fdf5295d62f4d41fd1f2121ba8f20250ab939d7e44e3dd8289dab29843e7"} Jul 28 20:00:37.544871 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.544184 2581 generic.go:356] "Generic (PLEG): container finished" podID="2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0" containerID="98445717b8b2dcfb7deb7991d9966330f45eb58d6c92b81dc8984f9086e8fc41" exitCode=0 Jul 28 20:00:37.544871 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.544226 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8kjr" event={"ID":"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0","Type":"ContainerDied","Data":"98445717b8b2dcfb7deb7991d9966330f45eb58d6c92b81dc8984f9086e8fc41"} Jul 28 20:00:37.555443 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.551249 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-v9k6d" podStartSLOduration=2.399519479 podStartE2EDuration="5.551227697s" totalImagesPullingTime="3.151708218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:37.548549577 +0000 UTC m=+64.992860596" watchObservedRunningTime="2026-07-28 20:00:37.551227697 +0000 UTC m=+64.995538714" Jul 28 20:00:37.583032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.582859 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5qckb" podStartSLOduration=2.70995084 podStartE2EDuration="5.58283919s" totalImagesPullingTime="2.87288835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:37.582462876 +0000 UTC m=+65.026773892" watchObservedRunningTime="2026-07-28 20:00:37.58283919 +0000 UTC m=+65.027150206" Jul 28 20:00:37.677221 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.677190 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7"] Jul 28 20:00:37.680412 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.680350 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:37.682663 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.682642 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vp4cg\"" Jul 28 20:00:37.682867 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.682851 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 20:00:37.691626 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.691599 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7"] Jul 28 20:00:37.691730 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.691697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc"] Jul 28 20:00:37.749266 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.749236 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpj4h\" (UniqueName: \"kubernetes.io/projected/a02098c4-4afc-461f-b380-3df3cc327c92-kube-api-access-xpj4h\") pod \"monitoring-plugin-68b7fc9574-fwvp7\" (UID: \"a02098c4-4afc-461f-b380-3df3cc327c92\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:37.749397 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.749299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a02098c4-4afc-461f-b380-3df3cc327c92-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fwvp7\" (UID: \"a02098c4-4afc-461f-b380-3df3cc327c92\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:37.850280 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.850238 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpj4h\" (UniqueName: \"kubernetes.io/projected/a02098c4-4afc-461f-b380-3df3cc327c92-kube-api-access-xpj4h\") pod \"monitoring-plugin-68b7fc9574-fwvp7\" (UID: \"a02098c4-4afc-461f-b380-3df3cc327c92\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:37.850774 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.850306 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a02098c4-4afc-461f-b380-3df3cc327c92-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fwvp7\" (UID: \"a02098c4-4afc-461f-b380-3df3cc327c92\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:37.850774 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:37.850488 2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 28 20:00:37.850774 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:00:37.850554 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02098c4-4afc-461f-b380-3df3cc327c92-monitoring-plugin-cert podName:a02098c4-4afc-461f-b380-3df3cc327c92 nodeName:}" failed. No retries permitted until 2026-07-28 20:00:38.350533586 +0000 UTC m=+65.794844584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a02098c4-4afc-461f-b380-3df3cc327c92-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-fwvp7" (UID: "a02098c4-4afc-461f-b380-3df3cc327c92") : secret "monitoring-plugin-cert" not found Jul 28 20:00:37.861314 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:37.861282 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpj4h\" (UniqueName: \"kubernetes.io/projected/a02098c4-4afc-461f-b380-3df3cc327c92-kube-api-access-xpj4h\") pod \"monitoring-plugin-68b7fc9574-fwvp7\" (UID: \"a02098c4-4afc-461f-b380-3df3cc327c92\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:38.355041 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.354953 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a02098c4-4afc-461f-b380-3df3cc327c92-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fwvp7\" (UID: \"a02098c4-4afc-461f-b380-3df3cc327c92\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:38.357957 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.357931 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a02098c4-4afc-461f-b380-3df3cc327c92-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fwvp7\" (UID: \"a02098c4-4afc-461f-b380-3df3cc327c92\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:38.549747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.549702 2581 generic.go:356] "Generic (PLEG): container finished" podID="62482197-a74a-4d25-8490-36f4559ec673" containerID="2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a" exitCode=0 Jul 28 20:00:38.549909 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.549802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a"} Jul 28 20:00:38.552336 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.552274 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8kjr" event={"ID":"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0","Type":"ContainerStarted","Data":"3392e4422e10cd18168340d13e8da11b9fdddcd7a9e3469835ac8afe174a3c11"} Jul 28 20:00:38.552514 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.552352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c8kjr" event={"ID":"2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0","Type":"ContainerStarted","Data":"742cccde50dbf35e254c77224137a3f3986b2e7b25c263dd93749b574d4b3296"} Jul 28 20:00:38.553645 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.553617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" event={"ID":"c62c6f2b-4184-4048-84f1-955575f8a2d9","Type":"ContainerStarted","Data":"7b35b78312b10aab101796a5934ad42ee7ec5c8a439dd3e2166a9972199a9c94"} Jul 28 20:00:38.591451 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.591401 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c8kjr" podStartSLOduration=3.240880074 podStartE2EDuration="6.591380222s" totalImagesPullingTime="3.244496377s" totalInitContainerRuntime="106.003771ms" isStatefulPod=true podCreationTimestamp="2026-07-28 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:38.590086522 +0000 UTC m=+66.034397538" watchObservedRunningTime="2026-07-28 20:00:38.591380222 +0000 UTC m=+66.035691237" Jul 28 20:00:38.632115 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.632022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:38.711155 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.711074 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7559966c56-zdbb4"] Jul 28 20:00:38.734259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.734153 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b78d555cb-bhg5x"] Jul 28 20:00:38.736969 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.736950 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.747470 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.747017 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b78d555cb-bhg5x"] Jul 28 20:00:38.759392 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.759252 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:38.761557 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.761469 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 20:00:38.772256 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.772233 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bca7392-4c0d-4e95-9e7c-3eb68914d111-metrics-certs\") pod \"network-metrics-daemon-xzqcj\" (UID: \"6bca7392-4c0d-4e95-9e7c-3eb68914d111\") " pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:38.786120 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.786098 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7"] Jul 28 20:00:38.861467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-config\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.861467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-oauth-serving-cert\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.861467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-oauth-config\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.861467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-service-ca\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.861467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-trusted-ca-bundle\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.862113 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-serving-cert\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.862113 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6kv\" (UniqueName: \"kubernetes.io/projected/32810a11-cfd4-4187-bf69-1c8779adbb7a-kube-api-access-sv6kv\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.862113 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.861719 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:38.863811 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.863763 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 20:00:38.874585 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.874559 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8f668389-c6ba-44b7-aded-ab82eb3cc122-original-pull-secret\") pod \"global-pull-secret-syncer-sq8gz\" (UID: \"8f668389-c6ba-44b7-aded-ab82eb3cc122\") " pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:38.963143 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-serving-cert\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.963143 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6kv\" (UniqueName: \"kubernetes.io/projected/32810a11-cfd4-4187-bf69-1c8779adbb7a-kube-api-access-sv6kv\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.963143 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:38.963395 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-config\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.963395 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963214 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-oauth-serving-cert\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.963512 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-oauth-config\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.963562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-service-ca\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.963593 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.963576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-trusted-ca-bundle\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.964182 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.964119 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-config\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.964182 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.964156 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-oauth-serving-cert\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.964416 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.964394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-service-ca\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.965026 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.964976 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-trusted-ca-bundle\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.965530 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.965506 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 20:00:38.965997 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.965976 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gjwl7\"" Jul 28 20:00:38.966527 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.966508 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-oauth-config\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.966597 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.966548 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-serving-cert\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.967251 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.967229 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sq8gz" Jul 28 20:00:38.973158 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.973140 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzqcj" Jul 28 20:00:38.976513 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.976493 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 20:00:38.981659 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.981636 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6kv\" (UniqueName: \"kubernetes.io/projected/32810a11-cfd4-4187-bf69-1c8779adbb7a-kube-api-access-sv6kv\") pod \"console-b78d555cb-bhg5x\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:38.986970 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:38.986944 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlmf\" (UniqueName: \"kubernetes.io/projected/f60c1e92-7837-41b8-9bf1-0bebd43c48c7-kube-api-access-nwlmf\") pod \"network-check-target-vk459\" (UID: \"f60c1e92-7837-41b8-9bf1-0bebd43c48c7\") " pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:39.050384 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:39.050302 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:39.259010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:39.258981 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kg9b\"" Jul 28 20:00:39.267747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:39.267713 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:39.305677 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:39.305646 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02098c4_4afc_461f_b380_3df3cc327c92.slice/crio-5379110223a6add6626780442de554bd6e7ea15d42a454ce107179c9172d20e6 WatchSource:0}: Error finding container 5379110223a6add6626780442de554bd6e7ea15d42a454ce107179c9172d20e6: Status 404 returned error can't find the container with id 5379110223a6add6626780442de554bd6e7ea15d42a454ce107179c9172d20e6 Jul 28 20:00:39.560110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:39.560022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" event={"ID":"a02098c4-4afc-461f-b380-3df3cc327c92","Type":"ContainerStarted","Data":"5379110223a6add6626780442de554bd6e7ea15d42a454ce107179c9172d20e6"} Jul 28 20:00:39.974869 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:39.972252 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vk459"] Jul 28 20:00:40.017450 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.012039 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b78d555cb-bhg5x"] Jul 28 20:00:40.043463 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.043429 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzqcj"] Jul 28 20:00:40.247259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.247226 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sq8gz"] Jul 28 20:00:40.250949 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:00:40.250917 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f668389_c6ba_44b7_aded_ab82eb3cc122.slice/crio-6be6c3b02d5b0f88e1762e99472c0bad05d419a72b36a61ddac2b2c775bbdf27 WatchSource:0}: Error finding container 6be6c3b02d5b0f88e1762e99472c0bad05d419a72b36a61ddac2b2c775bbdf27: Status 404 returned error can't find the container with id 6be6c3b02d5b0f88e1762e99472c0bad05d419a72b36a61ddac2b2c775bbdf27 Jul 28 20:00:40.566553 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.566511 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78d555cb-bhg5x" event={"ID":"32810a11-cfd4-4187-bf69-1c8779adbb7a","Type":"ContainerStarted","Data":"73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9"} Jul 28 20:00:40.566733 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.566568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78d555cb-bhg5x" event={"ID":"32810a11-cfd4-4187-bf69-1c8779adbb7a","Type":"ContainerStarted","Data":"f48a7188d7b82ae1f7fa4c295f451ec7a3d857b49cc9ca096a14e9d50882ce2e"} Jul 28 20:00:40.567910 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.567882 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzqcj" event={"ID":"6bca7392-4c0d-4e95-9e7c-3eb68914d111","Type":"ContainerStarted","Data":"596f6975ec67660c0e0c48e12c43b67a9a4b116a987af492c78772776f9ee6a9"} Jul 28 20:00:40.569224 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.569196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sq8gz" event={"ID":"8f668389-c6ba-44b7-aded-ab82eb3cc122","Type":"ContainerStarted","Data":"6be6c3b02d5b0f88e1762e99472c0bad05d419a72b36a61ddac2b2c775bbdf27"} Jul 28 20:00:40.570592 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.570523 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vk459" event={"ID":"f60c1e92-7837-41b8-9bf1-0bebd43c48c7","Type":"ContainerStarted","Data":"486ef0712226df8ce15951ae91aec08c44b9977d451e3621b401e88f494bbb1c"} Jul 28 20:00:40.573018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.572994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" event={"ID":"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78","Type":"ContainerStarted","Data":"bbde7df796fe49f52f3851b99c05e7dfb47d490f5394c587bd239516de3b82cb"} Jul 28 20:00:40.573106 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.573058 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" event={"ID":"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78","Type":"ContainerStarted","Data":"85cdb3fd5a93dc725aa7162119beea142abf344e825312c1edde9efaf335137e"} Jul 28 20:00:40.573157 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.573104 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" event={"ID":"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78","Type":"ContainerStarted","Data":"fe42693032b112e998e2e6e6b34e166e727bc9332649789e3d9388adecdecc3f"} Jul 28 20:00:40.574734 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.574689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" event={"ID":"c62c6f2b-4184-4048-84f1-955575f8a2d9","Type":"ContainerStarted","Data":"af4c31ac4af07181d1e9480eebf17c099d735eae3f93930618c717cd8d7a5f2f"} Jul 28 20:00:40.587011 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.586953 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b78d555cb-bhg5x" podStartSLOduration=2.586939251 podStartE2EDuration="2.586939251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:40.585218343 +0000 UTC m=+68.029529360" watchObservedRunningTime="2026-07-28 20:00:40.586939251 +0000 UTC m=+68.031250264" Jul 28 20:00:40.602962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:40.602912 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" podStartSLOduration=1.5436475170000001 podStartE2EDuration="3.602900095s" totalImagesPullingTime="2.059252578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:40.601831166 +0000 UTC m=+68.046142183" watchObservedRunningTime="2026-07-28 20:00:40.602900095 +0000 UTC m=+68.047211115" Jul 28 20:00:41.806663 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:41.806554 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:00:42.604664 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.604633 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerStarted","Data":"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0"} Jul 28 20:00:42.604813 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.604717 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerStarted","Data":"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050"} Jul 28 20:00:42.604813 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.604753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerStarted","Data":"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449"} Jul 28 20:00:42.604813 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.604794 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerStarted","Data":"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8"} Jul 28 20:00:42.605028 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.604832 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerStarted","Data":"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c"} Jul 28 20:00:42.606395 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.606367 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" event={"ID":"a02098c4-4afc-461f-b380-3df3cc327c92","Type":"ContainerStarted","Data":"8ca0a88b80ed1a437ac56bd0e4b287b51165de623d5cb1f509e1e49b987861e2"} Jul 28 20:00:42.606581 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.606562 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:42.609360 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.609313 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzqcj" event={"ID":"6bca7392-4c0d-4e95-9e7c-3eb68914d111","Type":"ContainerStarted","Data":"e7480934eebecfdabeba74b283fa4abdc1a36836cae9269e5ae37a78dbe111cc"} Jul 28 20:00:42.609482 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.609386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzqcj" event={"ID":"6bca7392-4c0d-4e95-9e7c-3eb68914d111","Type":"ContainerStarted","Data":"12090b3f31baf523e4edba204191e5d2f8776cb8f6bc830b4d9caa8854bdb091"} Jul 28 20:00:42.612839 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.612820 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" Jul 28 20:00:42.613428 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.613392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" event={"ID":"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78","Type":"ContainerStarted","Data":"834b00a8b3436e470ffb181f8cc4c647afb9a5be880735b5d4e1cf9643c77679"} Jul 28 20:00:42.613528 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.613449 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" event={"ID":"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78","Type":"ContainerStarted","Data":"a078c8bca906506208dc99943496b4b0c2ac21004353609828de2c01d39940e9"} Jul 28 20:00:42.622571 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.622520 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fwvp7" podStartSLOduration=3.742967227 podStartE2EDuration="5.622499902s" totalImagesPullingTime="1.879532675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:42.620490592 +0000 UTC m=+70.064801609" watchObservedRunningTime="2026-07-28 20:00:42.622499902 +0000 UTC m=+70.066810898" Jul 28 20:00:42.653527 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:42.653401 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xzqcj" podStartSLOduration=68.208545774 podStartE2EDuration="1m9.653384608s" totalImagesPullingTime="1.444838834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:42.653205998 +0000 UTC m=+70.097517027" watchObservedRunningTime="2026-07-28 20:00:42.653384608 +0000 UTC m=+70.097695621" Jul 28 20:00:43.623222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:43.623171 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" event={"ID":"c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78","Type":"ContainerStarted","Data":"688be30fca15181ef8084f1b2cbd06115fd4a55cd6e5e49a339e6dd793046649"} Jul 28 20:00:43.623943 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:43.623873 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:43.647782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:43.647715 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" podStartSLOduration=4.48199469 podStartE2EDuration="9.64769279s" totalImagesPullingTime="5.1656981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:43.645547813 +0000 UTC m=+71.089858830" watchObservedRunningTime="2026-07-28 20:00:43.64769279 +0000 UTC m=+71.092003807" Jul 28 20:00:45.635670 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:45.635632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerStarted","Data":"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5"} Jul 28 20:00:45.637179 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:45.637153 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sq8gz" event={"ID":"8f668389-c6ba-44b7-aded-ab82eb3cc122","Type":"ContainerStarted","Data":"d06e049c369208f1f5d7e51d1d8e57b2d12878663d5757ba0ba9da9cca1011ec"} Jul 28 20:00:45.638459 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:45.638439 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vk459" event={"ID":"f60c1e92-7837-41b8-9bf1-0bebd43c48c7","Type":"ContainerStarted","Data":"7f13061a5836641bd04f9f6fd69381f316ccd3ddb871b23b3c5f86509ceeee91"} Jul 28 20:00:45.638566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:45.638551 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:00:45.660534 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:45.660492 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.764828856 podStartE2EDuration="12.660478195s" totalImagesPullingTime="6.874212992s" totalInitContainerRuntime="21.436347ms" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:45.659272009 +0000 UTC m=+73.103583025" watchObservedRunningTime="2026-07-28 20:00:45.660478195 +0000 UTC m=+73.104789210" Jul 28 20:00:45.680069 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:45.680010 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sq8gz" podStartSLOduration=67.781393644 podStartE2EDuration="1m12.679987824s" totalImagesPullingTime="4.89859418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:45.679470103 +0000 UTC m=+73.123781119" watchObservedRunningTime="2026-07-28 20:00:45.679987824 +0000 UTC m=+73.124298840" Jul 28 20:00:45.696018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:45.695960 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vk459" podStartSLOduration=67.529608032 podStartE2EDuration="1m12.695940762s" totalImagesPullingTime="5.16633273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:45.694432909 +0000 UTC m=+73.138743922" watchObservedRunningTime="2026-07-28 20:00:45.695940762 +0000 UTC m=+73.140251779" Jul 28 20:00:49.050408 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:49.050367 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:49.050804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:49.050439 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:49.055306 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:49.055284 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:49.634104 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:49.634077 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-54bbc586dd-qj699" Jul 28 20:00:49.661626 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:49.661597 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:00:49.705880 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:49.705840 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dc84b9d7-7vzzf"] Jul 28 20:00:57.489668 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:57.489549 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:00:57.489668 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:00:57.489606 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:01:03.731822 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:03.731760 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7559966c56-zdbb4" podUID="d4695943-4ed7-4890-b92a-fa1fc0ce11ca" containerName="console" containerID="cri-o://f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308" gracePeriod=15 Jul 28 20:01:04.611746 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.611719 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:01:04.717491 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.717406 2581 generic.go:356] "Generic (PLEG): container finished" podID="d4695943-4ed7-4890-b92a-fa1fc0ce11ca" containerID="f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308" exitCode=0 Jul 28 20:01:04.717491 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.717451 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7559966c56-zdbb4" event={"ID":"d4695943-4ed7-4890-b92a-fa1fc0ce11ca","Type":"ContainerDied","Data":"f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308"} Jul 28 20:01:04.717721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.717453 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7559966c56-zdbb4" Jul 28 20:01:04.717721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.717534 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7559966c56-zdbb4" event={"ID":"d4695943-4ed7-4890-b92a-fa1fc0ce11ca","Type":"ContainerDied","Data":"fe02f19bc6e819e344cc3d0362d62abbaf06924765b3fdf507370d98ba453bf3"} Jul 28 20:01:04.717721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.717563 2581 scope.go:117] "RemoveContainer" containerID="f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308" Jul 28 20:01:04.725553 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.725535 2581 scope.go:117] "RemoveContainer" containerID="f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308" Jul 28 20:01:04.725818 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:04.725797 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308\": container with ID starting with f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308 not found: ID does not exist" containerID="f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308" Jul 28 20:01:04.725872 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.725831 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308"} err="failed to get container status \"f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308\": rpc error: code = NotFound desc = could not find container \"f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308\": container with ID starting with f63f9bc42e1baae474e8cdaadbf17af489a6296065ba336b04b52acab19cb308 not found: ID does not exist" Jul 28 20:01:04.806259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806223 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-oauth-config\") pod \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " Jul 28 20:01:04.806680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806274 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-service-ca\") pod \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " Jul 28 20:01:04.806680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806361 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-trusted-ca-bundle\") pod \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " Jul 28 20:01:04.806680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806389 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlmlr\" (UniqueName: \"kubernetes.io/projected/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-kube-api-access-qlmlr\") pod \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " Jul 28 20:01:04.806680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806409 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-oauth-serving-cert\") pod \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " Jul 28 20:01:04.806680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806441 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-config\") pod \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " Jul 28 20:01:04.806680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806457 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-serving-cert\") pod \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\" (UID: \"d4695943-4ed7-4890-b92a-fa1fc0ce11ca\") " Jul 28 20:01:04.807045 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806840 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-service-ca" (OuterVolumeSpecName: "service-ca") pod "d4695943-4ed7-4890-b92a-fa1fc0ce11ca" (UID: "d4695943-4ed7-4890-b92a-fa1fc0ce11ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:04.807045 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806875 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d4695943-4ed7-4890-b92a-fa1fc0ce11ca" (UID: "d4695943-4ed7-4890-b92a-fa1fc0ce11ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:04.807045 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806914 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d4695943-4ed7-4890-b92a-fa1fc0ce11ca" (UID: "d4695943-4ed7-4890-b92a-fa1fc0ce11ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:04.807045 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.806972 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-config" (OuterVolumeSpecName: "console-config") pod "d4695943-4ed7-4890-b92a-fa1fc0ce11ca" (UID: "d4695943-4ed7-4890-b92a-fa1fc0ce11ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:04.808857 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.808828 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d4695943-4ed7-4890-b92a-fa1fc0ce11ca" (UID: "d4695943-4ed7-4890-b92a-fa1fc0ce11ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:04.808857 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.808835 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-kube-api-access-qlmlr" (OuterVolumeSpecName: "kube-api-access-qlmlr") pod "d4695943-4ed7-4890-b92a-fa1fc0ce11ca" (UID: "d4695943-4ed7-4890-b92a-fa1fc0ce11ca"). InnerVolumeSpecName "kube-api-access-qlmlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:01:04.809028 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.808888 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d4695943-4ed7-4890-b92a-fa1fc0ce11ca" (UID: "d4695943-4ed7-4890-b92a-fa1fc0ce11ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:04.907865 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.907831 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-trusted-ca-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:04.907865 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.907858 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmlr\" (UniqueName: \"kubernetes.io/projected/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-kube-api-access-qlmlr\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:04.907865 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.907867 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-oauth-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:04.907865 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.907876 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:04.908114 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.907887 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:04.908114 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.907896 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-console-oauth-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:04.908114 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:04.907906 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4695943-4ed7-4890-b92a-fa1fc0ce11ca-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:05.036419 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:05.036389 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7559966c56-zdbb4"] Jul 28 20:01:05.038648 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:05.038627 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7559966c56-zdbb4"] Jul 28 20:01:05.151258 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:05.151223 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4695943-4ed7-4890-b92a-fa1fc0ce11ca" path="/var/lib/kubelet/pods/d4695943-4ed7-4890-b92a-fa1fc0ce11ca/volumes" Jul 28 20:01:14.725672 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:14.725627 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86dc84b9d7-7vzzf" podUID="2dcb0a3a-c07c-459e-ab58-3c31ca47062d" containerName="console" containerID="cri-o://7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc" gracePeriod=15 Jul 28 20:01:15.022871 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.022834 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:01:15.079288 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079254 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-config\") pod \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " Jul 28 20:01:15.079288 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079299 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-service-ca\") pod \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " Jul 28 20:01:15.079547 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079357 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-oauth-config\") pod \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " Jul 28 20:01:15.079547 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079388 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rsr\" (UniqueName: \"kubernetes.io/projected/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-kube-api-access-52rsr\") pod \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " Jul 28 20:01:15.079547 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-serving-cert\") pod \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " Jul 28 20:01:15.079547 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079451 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-oauth-serving-cert\") pod \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\" (UID: \"2dcb0a3a-c07c-459e-ab58-3c31ca47062d\") " Jul 28 20:01:15.079876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079840 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-service-ca" (OuterVolumeSpecName: "service-ca") pod "2dcb0a3a-c07c-459e-ab58-3c31ca47062d" (UID: "2dcb0a3a-c07c-459e-ab58-3c31ca47062d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:15.079989 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079769 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-config" (OuterVolumeSpecName: "console-config") pod "2dcb0a3a-c07c-459e-ab58-3c31ca47062d" (UID: "2dcb0a3a-c07c-459e-ab58-3c31ca47062d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:15.079989 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.079907 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2dcb0a3a-c07c-459e-ab58-3c31ca47062d" (UID: "2dcb0a3a-c07c-459e-ab58-3c31ca47062d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:15.081586 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.081564 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2dcb0a3a-c07c-459e-ab58-3c31ca47062d" (UID: "2dcb0a3a-c07c-459e-ab58-3c31ca47062d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:15.082025 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.082002 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2dcb0a3a-c07c-459e-ab58-3c31ca47062d" (UID: "2dcb0a3a-c07c-459e-ab58-3c31ca47062d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:15.082099 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.082025 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-kube-api-access-52rsr" (OuterVolumeSpecName: "kube-api-access-52rsr") pod "2dcb0a3a-c07c-459e-ab58-3c31ca47062d" (UID: "2dcb0a3a-c07c-459e-ab58-3c31ca47062d"). InnerVolumeSpecName "kube-api-access-52rsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:01:15.180024 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.179988 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:15.180024 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.180016 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-oauth-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:15.180024 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.180029 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:15.180250 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.180041 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:15.180250 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.180052 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-console-oauth-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:15.180250 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.180063 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rsr\" (UniqueName: \"kubernetes.io/projected/2dcb0a3a-c07c-459e-ab58-3c31ca47062d-kube-api-access-52rsr\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:15.763448 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.763411 2581 generic.go:356] "Generic (PLEG): container finished" podID="2dcb0a3a-c07c-459e-ab58-3c31ca47062d" containerID="7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc" exitCode=0 Jul 28 20:01:15.763854 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.763454 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dc84b9d7-7vzzf" Jul 28 20:01:15.763854 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.763494 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc84b9d7-7vzzf" event={"ID":"2dcb0a3a-c07c-459e-ab58-3c31ca47062d","Type":"ContainerDied","Data":"7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc"} Jul 28 20:01:15.763854 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.763541 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dc84b9d7-7vzzf" event={"ID":"2dcb0a3a-c07c-459e-ab58-3c31ca47062d","Type":"ContainerDied","Data":"f8ee3b51724ba4f3810c4a4e8d4ef1be8c9703388d14144fcd8a20e47a2d708b"} Jul 28 20:01:15.763854 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.763562 2581 scope.go:117] "RemoveContainer" containerID="7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc" Jul 28 20:01:15.771836 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.771819 2581 scope.go:117] "RemoveContainer" containerID="7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc" Jul 28 20:01:15.772094 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:15.772075 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc\": container with ID starting with 7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc not found: ID does not exist" containerID="7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc" Jul 28 20:01:15.772141 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.772103 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc"} err="failed to get container status \"7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc\": rpc error: code = NotFound desc = could not find container \"7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc\": container with ID starting with 7f65c61913661aa0357680f0a11452852ab1da188ae4daa1ea878c3615dc3cdc not found: ID does not exist" Jul 28 20:01:15.784711 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.784683 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dc84b9d7-7vzzf"] Jul 28 20:01:15.792338 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:15.792298 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86dc84b9d7-7vzzf"] Jul 28 20:01:16.645245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:16.645217 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vk459" Jul 28 20:01:17.151416 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:17.151383 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dcb0a3a-c07c-459e-ab58-3c31ca47062d" path="/var/lib/kubelet/pods/2dcb0a3a-c07c-459e-ab58-3c31ca47062d/volumes" Jul 28 20:01:17.495591 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:17.495562 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:01:17.499311 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:17.499293 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bc6fdb7cf-v75qc" Jul 28 20:01:53.184224 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.184183 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:01:53.184752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.184476 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="alertmanager" containerID="cri-o://ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0" gracePeriod=120 Jul 28 20:01:53.184752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.184533 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-metric" containerID="cri-o://157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050" gracePeriod=120 Jul 28 20:01:53.184752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.184566 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-web" containerID="cri-o://637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8" gracePeriod=120 Jul 28 20:01:53.184752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.184591 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="config-reloader" containerID="cri-o://b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c" gracePeriod=120 Jul 28 20:01:53.184752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.184652 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="prom-label-proxy" containerID="cri-o://76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5" gracePeriod=120 Jul 28 20:01:53.184752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.184614 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy" containerID="cri-o://73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449" gracePeriod=120 Jul 28 20:01:53.916566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916533 2581 generic.go:356] "Generic (PLEG): container finished" podID="62482197-a74a-4d25-8490-36f4559ec673" containerID="76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5" exitCode=0 Jul 28 20:01:53.916566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916560 2581 generic.go:356] "Generic (PLEG): container finished" podID="62482197-a74a-4d25-8490-36f4559ec673" containerID="73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449" exitCode=0 Jul 28 20:01:53.916566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916567 2581 generic.go:356] "Generic (PLEG): container finished" podID="62482197-a74a-4d25-8490-36f4559ec673" containerID="b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c" exitCode=0 Jul 28 20:01:53.916566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916572 2581 generic.go:356] "Generic (PLEG): container finished" podID="62482197-a74a-4d25-8490-36f4559ec673" containerID="ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0" exitCode=0 Jul 28 20:01:53.916848 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916595 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5"} Jul 28 20:01:53.916848 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916657 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449"} Jul 28 20:01:53.916848 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916686 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c"} Jul 28 20:01:53.916848 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:53.916710 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0"} Jul 28 20:01:54.427632 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.427608 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:54.524586 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524485 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524586 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524531 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-web-config\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524586 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524571 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdrmq\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-kube-api-access-kdrmq\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524612 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524649 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-config-out\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524670 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-cluster-tls-config\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524713 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-config-volume\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524743 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-tls-assets\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524781 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524809 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-metrics-client-ca\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.524847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524840 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-metric\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.525376 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524869 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-main-db\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.525376 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.524908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-web\") pod \"62482197-a74a-4d25-8490-36f4559ec673\" (UID: \"62482197-a74a-4d25-8490-36f4559ec673\") " Jul 28 20:01:54.526069 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.526029 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:01:54.526435 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.526405 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:54.527092 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.526804 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:01:54.528919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.528881 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:54.529854 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.529828 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:54.530204 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.530151 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-kube-api-access-kdrmq" (OuterVolumeSpecName: "kube-api-access-kdrmq") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "kube-api-access-kdrmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:01:54.530204 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.530179 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-config-out" (OuterVolumeSpecName: "config-out") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:01:54.530400 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.530301 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:54.530898 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.530870 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-config-volume" (OuterVolumeSpecName: "config-volume") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:54.530970 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.530954 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:54.531177 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.531159 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:01:54.534852 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.534830 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:54.540904 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.540883 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-web-config" (OuterVolumeSpecName: "web-config") pod "62482197-a74a-4d25-8490-36f4559ec673" (UID: "62482197-a74a-4d25-8490-36f4559ec673"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:01:54.625603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625555 2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625596 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-tls-assets\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625605 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625616 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-metrics-client-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625627 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625636 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-main-db\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625645 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625654 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-secret-alertmanager-main-tls\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625662 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-web-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625670 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdrmq\" (UniqueName: \"kubernetes.io/projected/62482197-a74a-4d25-8490-36f4559ec673-kube-api-access-kdrmq\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625679 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62482197-a74a-4d25-8490-36f4559ec673-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625687 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62482197-a74a-4d25-8490-36f4559ec673-config-out\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.625828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.625695 2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/62482197-a74a-4d25-8490-36f4559ec673-cluster-tls-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:01:54.928530 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.928443 2581 generic.go:356] "Generic (PLEG): container finished" podID="62482197-a74a-4d25-8490-36f4559ec673" containerID="157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050" exitCode=0 Jul 28 20:01:54.928530 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.928473 2581 generic.go:356] "Generic (PLEG): container finished" podID="62482197-a74a-4d25-8490-36f4559ec673" containerID="637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8" exitCode=0 Jul 28 20:01:54.928530 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.928496 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050"} Jul 28 20:01:54.928530 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.928523 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:54.928764 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.928587 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8"} Jul 28 20:01:54.928764 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.928615 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"62482197-a74a-4d25-8490-36f4559ec673","Type":"ContainerDied","Data":"c8c3fdf5295d62f4d41fd1f2121ba8f20250ab939d7e44e3dd8289dab29843e7"} Jul 28 20:01:54.928764 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.928645 2581 scope.go:117] "RemoveContainer" containerID="76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5" Jul 28 20:01:54.936502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.936478 2581 scope.go:117] "RemoveContainer" containerID="157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050" Jul 28 20:01:54.944375 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.944354 2581 scope.go:117] "RemoveContainer" containerID="73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449" Jul 28 20:01:54.951674 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.951644 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:01:54.952538 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.951958 2581 scope.go:117] "RemoveContainer" containerID="637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8" Jul 28 20:01:54.957643 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.957621 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:01:54.960381 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.960357 2581 scope.go:117] "RemoveContainer" containerID="b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c" Jul 28 20:01:54.967144 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.967127 2581 scope.go:117] "RemoveContainer" containerID="ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0" Jul 28 20:01:54.974358 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.974336 2581 scope.go:117] "RemoveContainer" containerID="2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a" Jul 28 20:01:54.981391 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.981374 2581 scope.go:117] "RemoveContainer" containerID="76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5" Jul 28 20:01:54.981653 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:54.981635 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5\": container with ID starting with 76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5 not found: ID does not exist" containerID="76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5" Jul 28 20:01:54.981721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.981661 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5"} err="failed to get container status \"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5\": rpc error: code = NotFound desc = could not find container \"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5\": container with ID starting with 76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5 not found: ID does not exist" Jul 28 20:01:54.981721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.981681 2581 scope.go:117] "RemoveContainer" containerID="157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050" Jul 28 20:01:54.981920 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:54.981903 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050\": container with ID starting with 157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050 not found: ID does not exist" containerID="157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050" Jul 28 20:01:54.981964 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.981926 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050"} err="failed to get container status \"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050\": rpc error: code = NotFound desc = could not find container \"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050\": container with ID starting with 157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050 not found: ID does not exist" Jul 28 20:01:54.981964 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.981942 2581 scope.go:117] "RemoveContainer" containerID="73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449" Jul 28 20:01:54.982170 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:54.982154 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449\": container with ID starting with 73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449 not found: ID does not exist" containerID="73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449" Jul 28 20:01:54.982227 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982172 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449"} err="failed to get container status \"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449\": rpc error: code = NotFound desc = could not find container \"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449\": container with ID starting with 73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449 not found: ID does not exist" Jul 28 20:01:54.982227 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982183 2581 scope.go:117] "RemoveContainer" containerID="637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8" Jul 28 20:01:54.982422 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:54.982406 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8\": container with ID starting with 637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8 not found: ID does not exist" containerID="637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8" Jul 28 20:01:54.982471 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982426 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8"} err="failed to get container status \"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8\": rpc error: code = NotFound desc = could not find container \"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8\": container with ID starting with 637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8 not found: ID does not exist" Jul 28 20:01:54.982471 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982439 2581 scope.go:117] "RemoveContainer" containerID="b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c" Jul 28 20:01:54.982668 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:54.982651 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c\": container with ID starting with b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c not found: ID does not exist" containerID="b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c" Jul 28 20:01:54.982713 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982672 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c"} err="failed to get container status \"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c\": rpc error: code = NotFound desc = could not find container \"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c\": container with ID starting with b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c not found: ID does not exist" Jul 28 20:01:54.982713 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982686 2581 scope.go:117] "RemoveContainer" containerID="ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0" Jul 28 20:01:54.982888 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:54.982874 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0\": container with ID starting with ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0 not found: ID does not exist" containerID="ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0" Jul 28 20:01:54.982926 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982895 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0"} err="failed to get container status \"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0\": rpc error: code = NotFound desc = could not find container \"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0\": container with ID starting with ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0 not found: ID does not exist" Jul 28 20:01:54.982926 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.982906 2581 scope.go:117] "RemoveContainer" containerID="2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a" Jul 28 20:01:54.983108 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:01:54.983093 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a\": container with ID starting with 2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a not found: ID does not exist" containerID="2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a" Jul 28 20:01:54.983146 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983111 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a"} err="failed to get container status \"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a\": rpc error: code = NotFound desc = could not find container \"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a\": container with ID starting with 2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a not found: ID does not exist" Jul 28 20:01:54.983146 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983124 2581 scope.go:117] "RemoveContainer" containerID="76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5" Jul 28 20:01:54.983313 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983298 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5"} err="failed to get container status \"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5\": rpc error: code = NotFound desc = could not find container \"76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5\": container with ID starting with 76c3b35b24a3b249baeae2b6f0ee7ef4250f46c4d62f1e6e52d7c986430be5c5 not found: ID does not exist" Jul 28 20:01:54.983368 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983313 2581 scope.go:117] "RemoveContainer" containerID="157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050" Jul 28 20:01:54.983506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983491 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050"} err="failed to get container status \"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050\": rpc error: code = NotFound desc = could not find container \"157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050\": container with ID starting with 157fdc7df1959b2939379e1b9e54055222ec34cbcf4fc3a39a229a3c2df56050 not found: ID does not exist" Jul 28 20:01:54.983547 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983506 2581 scope.go:117] "RemoveContainer" containerID="73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449" Jul 28 20:01:54.983692 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983662 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449"} err="failed to get container status \"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449\": rpc error: code = NotFound desc = could not find container \"73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449\": container with ID starting with 73290e6deba13040e5798f6663308a90714256fd30111bec021ea31c8b5f1449 not found: ID does not exist" Jul 28 20:01:54.983692 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983690 2581 scope.go:117] "RemoveContainer" containerID="637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8" Jul 28 20:01:54.983924 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983908 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8"} err="failed to get container status \"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8\": rpc error: code = NotFound desc = could not find container \"637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8\": container with ID starting with 637051a6ff64aeb20fe957a0e59b862ab60c65e50d7169edb019e29d2f9471a8 not found: ID does not exist" Jul 28 20:01:54.983979 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.983925 2581 scope.go:117] "RemoveContainer" containerID="b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c" Jul 28 20:01:54.984133 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.984115 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c"} err="failed to get container status \"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c\": rpc error: code = NotFound desc = could not find container \"b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c\": container with ID starting with b8b9740bcb321006192c5679435319f6f1eace99ab67ed5be3922ef504051f5c not found: ID does not exist" Jul 28 20:01:54.984181 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.984133 2581 scope.go:117] "RemoveContainer" containerID="ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0" Jul 28 20:01:54.984346 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.984309 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0"} err="failed to get container status \"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0\": rpc error: code = NotFound desc = could not find container \"ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0\": container with ID starting with ea05f9705461825298c25eaa7794ac1862300e0cb7c574196bc8269fd999e0f0 not found: ID does not exist" Jul 28 20:01:54.984418 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.984346 2581 scope.go:117] "RemoveContainer" containerID="2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a" Jul 28 20:01:54.984556 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.984541 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a"} err="failed to get container status \"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a\": rpc error: code = NotFound desc = could not find container \"2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a\": container with ID starting with 2b64603c45e9e3c3cc84676f6600cd0e262a5cc0e8d966352b29b157f9bdfc8a not found: ID does not exist" Jul 28 20:01:54.987613 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987589 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:01:54.987908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987892 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcb0a3a-c07c-459e-ab58-3c31ca47062d" containerName="console" Jul 28 20:01:54.987978 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987911 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcb0a3a-c07c-459e-ab58-3c31ca47062d" containerName="console" Jul 28 20:01:54.987978 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987927 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="init-config-reloader" Jul 28 20:01:54.987978 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987935 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="init-config-reloader" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987978 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="alertmanager" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987988 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="alertmanager" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.987999 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="prom-label-proxy" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988007 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="prom-label-proxy" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988019 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="config-reloader" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988028 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="config-reloader" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988041 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4695943-4ed7-4890-b92a-fa1fc0ce11ca" containerName="console" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988050 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4695943-4ed7-4890-b92a-fa1fc0ce11ca" containerName="console" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988059 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-web" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988068 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-web" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988077 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988084 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988109 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-metric" Jul 28 20:01:54.988130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988118 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-metric" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988200 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="prom-label-proxy" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988214 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-metric" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988225 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4695943-4ed7-4890-b92a-fa1fc0ce11ca" containerName="console" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988235 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="config-reloader" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988244 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy-web" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988253 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcb0a3a-c07c-459e-ab58-3c31ca47062d" containerName="console" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988262 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="alertmanager" Jul 28 20:01:54.988788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.988270 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62482197-a74a-4d25-8490-36f4559ec673" containerName="kube-rbac-proxy" Jul 28 20:01:54.993810 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.993789 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:54.995754 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.995733 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 20:01:54.995860 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.995757 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 20:01:54.995925 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.995897 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 20:01:54.996063 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.996046 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-52z6n\"" Jul 28 20:01:54.996120 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.996095 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 20:01:54.996120 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.996101 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 20:01:54.996213 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.996052 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 20:01:54.996258 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.996240 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 20:01:54.996396 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:54.996379 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 20:01:55.003087 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.003066 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 20:01:55.006245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.006228 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:01:55.129289 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce04cd2d-19af-411d-9316-50c8cb949b5a-config-out\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129289 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129287 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-web-config\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04cd2d-19af-411d-9316-50c8cb949b5a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce04cd2d-19af-411d-9316-50c8cb949b5a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129440 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmrsn\" (UniqueName: \"kubernetes.io/projected/ce04cd2d-19af-411d-9316-50c8cb949b5a-kube-api-access-hmrsn\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce04cd2d-19af-411d-9316-50c8cb949b5a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129499 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce04cd2d-19af-411d-9316-50c8cb949b5a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.129906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.129697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.150839 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.150808 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62482197-a74a-4d25-8490-36f4559ec673" path="/var/lib/kubelet/pods/62482197-a74a-4d25-8490-36f4559ec673/volumes" Jul 28 20:01:55.230357 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230357 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230367 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce04cd2d-19af-411d-9316-50c8cb949b5a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230571 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce04cd2d-19af-411d-9316-50c8cb949b5a-config-out\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230599 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-web-config\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230636 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04cd2d-19af-411d-9316-50c8cb949b5a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce04cd2d-19af-411d-9316-50c8cb949b5a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrsn\" (UniqueName: \"kubernetes.io/projected/ce04cd2d-19af-411d-9316-50c8cb949b5a-kube-api-access-hmrsn\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.230804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce04cd2d-19af-411d-9316-50c8cb949b5a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.231162 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.230907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce04cd2d-19af-411d-9316-50c8cb949b5a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233116 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.231435 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce04cd2d-19af-411d-9316-50c8cb949b5a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233116 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.231729 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce04cd2d-19af-411d-9316-50c8cb949b5a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233283 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.233207 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce04cd2d-19af-411d-9316-50c8cb949b5a-config-out\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233448 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.233423 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.233529 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233649 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.233631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233703 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.233681 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233877 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.233855 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.233877 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.233865 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce04cd2d-19af-411d-9316-50c8cb949b5a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.234174 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.234154 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-web-config\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.235110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.235093 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce04cd2d-19af-411d-9316-50c8cb949b5a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.240548 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.240530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrsn\" (UniqueName: \"kubernetes.io/projected/ce04cd2d-19af-411d-9316-50c8cb949b5a-kube-api-access-hmrsn\") pod \"alertmanager-main-0\" (UID: \"ce04cd2d-19af-411d-9316-50c8cb949b5a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.303720 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.303686 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 20:01:55.438988 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.438963 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 20:01:55.441501 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:01:55.441473 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce04cd2d_19af_411d_9316_50c8cb949b5a.slice/crio-0376d3b9d72354a2b571d893c4823c5086311a8ec611d6ea2b084cd5dda564aa WatchSource:0}: Error finding container 0376d3b9d72354a2b571d893c4823c5086311a8ec611d6ea2b084cd5dda564aa: Status 404 returned error can't find the container with id 0376d3b9d72354a2b571d893c4823c5086311a8ec611d6ea2b084cd5dda564aa Jul 28 20:01:55.934750 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.934717 2581 generic.go:356] "Generic (PLEG): container finished" podID="ce04cd2d-19af-411d-9316-50c8cb949b5a" containerID="68be2c19432595471074eb333c06b2e382fd9c688ecaf54af99acfb9ddb8c094" exitCode=0 Jul 28 20:01:55.934933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.934787 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerDied","Data":"68be2c19432595471074eb333c06b2e382fd9c688ecaf54af99acfb9ddb8c094"} Jul 28 20:01:55.934933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:55.934837 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerStarted","Data":"0376d3b9d72354a2b571d893c4823c5086311a8ec611d6ea2b084cd5dda564aa"} Jul 28 20:01:56.942536 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:56.942494 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerStarted","Data":"69b795297b28ac6e9c106ae1d810f50222aa0fed75f0830b5f0d8c737d18c1c9"} Jul 28 20:01:56.942919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:56.942602 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerStarted","Data":"a278a4bf1f4c4e34f33551145491a18b3c990ea3474985d5b5e5327efe693139"} Jul 28 20:01:56.942919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:56.942630 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerStarted","Data":"80c72925711a31b78499887d5cb2faf90dd377e51d4bef68b3d82c89fa9fbb98"} Jul 28 20:01:56.942919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:56.942654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerStarted","Data":"e12b18d593281538dbbd71fb4471328e19c6c8fea1bd4545cc375b8320392c64"} Jul 28 20:01:56.942919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:56.942681 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerStarted","Data":"390cd1a2c8e6174e556246fc142c49b28a583fb12a9acea2ecde42bbae531f9c"} Jul 28 20:01:56.942919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:56.942705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce04cd2d-19af-411d-9316-50c8cb949b5a","Type":"ContainerStarted","Data":"8abc0f8ad48d5148e9b9518e0aef389685c2f9bbde85ef65bb7aa7f10c3c9ee4"} Jul 28 20:01:56.968163 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:56.968099 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.953962956 podStartE2EDuration="2.968078154s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.115198ms" isStatefulPod=false podCreationTimestamp="2026-07-28 20:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:01:56.966131628 +0000 UTC m=+144.410442644" watchObservedRunningTime="2026-07-28 20:01:56.968078154 +0000 UTC m=+144.412389170" Jul 28 20:01:57.222131 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.222042 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-dd8bb8f5-hh429"] Jul 28 20:01:57.225101 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.225078 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.227243 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.227213 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 20:01:57.227812 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.227783 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 20:01:57.228120 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.228098 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rhwk4\"" Jul 28 20:01:57.228308 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.228290 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 20:01:57.228508 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.228492 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 20:01:57.228705 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.228679 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 20:01:57.233874 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.233848 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 20:01:57.243779 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.243709 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dd8bb8f5-hh429"] Jul 28 20:01:57.349559 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-serving-certs-ca-bundle\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.349726 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.349726 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.349726 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349679 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-metrics-client-ca\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.349904 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349761 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-telemeter-client-tls\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.349904 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349790 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-federate-client-tls\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.349904 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5gc\" (UniqueName: \"kubernetes.io/projected/c586b467-e767-4d53-bc94-da07eac07192-kube-api-access-qc5gc\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.349904 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.349893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-secret-telemeter-client\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450281 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450238 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450281 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-metrics-client-ca\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-telemeter-client-tls\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-federate-client-tls\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5gc\" (UniqueName: \"kubernetes.io/projected/c586b467-e767-4d53-bc94-da07eac07192-kube-api-access-qc5gc\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-secret-telemeter-client\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.450572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.450541 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-serving-certs-ca-bundle\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.451241 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.451216 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-serving-certs-ca-bundle\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.451401 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.451377 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-metrics-client-ca\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.451461 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.451445 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c586b467-e767-4d53-bc94-da07eac07192-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.453417 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.453397 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-federate-client-tls\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.453515 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.453421 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.453638 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.453616 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-telemeter-client-tls\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.453699 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.453624 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c586b467-e767-4d53-bc94-da07eac07192-secret-telemeter-client\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.459098 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.459078 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5gc\" (UniqueName: \"kubernetes.io/projected/c586b467-e767-4d53-bc94-da07eac07192-kube-api-access-qc5gc\") pod \"telemeter-client-dd8bb8f5-hh429\" (UID: \"c586b467-e767-4d53-bc94-da07eac07192\") " pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.535942 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.535893 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" Jul 28 20:01:57.697192 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.697154 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dd8bb8f5-hh429"] Jul 28 20:01:57.699362 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:01:57.699317 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc586b467_e767_4d53_bc94_da07eac07192.slice/crio-16ad7a641fe8f6e3a57e3cb7d4a4d626649a72869adeeffbb533ed712431ca07 WatchSource:0}: Error finding container 16ad7a641fe8f6e3a57e3cb7d4a4d626649a72869adeeffbb533ed712431ca07: Status 404 returned error can't find the container with id 16ad7a641fe8f6e3a57e3cb7d4a4d626649a72869adeeffbb533ed712431ca07 Jul 28 20:01:57.947811 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:57.947707 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" event={"ID":"c586b467-e767-4d53-bc94-da07eac07192","Type":"ContainerStarted","Data":"16ad7a641fe8f6e3a57e3cb7d4a4d626649a72869adeeffbb533ed712431ca07"} Jul 28 20:01:59.957902 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:59.957808 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" event={"ID":"c586b467-e767-4d53-bc94-da07eac07192","Type":"ContainerStarted","Data":"4b371a0324260a46260f2f263035fd5bc5601a92d8c4479ed9167239b5e490c5"} Jul 28 20:01:59.957902 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:59.957878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" event={"ID":"c586b467-e767-4d53-bc94-da07eac07192","Type":"ContainerStarted","Data":"0eb213ca1f14431446edec1d1e03fcd8053d6ab3697fa492dcdcbd771b52f324"} Jul 28 20:01:59.958352 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:59.957909 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" event={"ID":"c586b467-e767-4d53-bc94-da07eac07192","Type":"ContainerStarted","Data":"afbe5a8e52d39f1dcb25b266fe71b49434ea281620a31b79599c355a39c9f574"} Jul 28 20:01:59.987747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:01:59.987690 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-dd8bb8f5-hh429" podStartSLOduration=1.154778092 podStartE2EDuration="2.987673654s" totalImagesPullingTime="1.832895562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:01:59.987106528 +0000 UTC m=+147.431417544" watchObservedRunningTime="2026-07-28 20:01:59.987673654 +0000 UTC m=+147.431984670" Jul 28 20:02:00.866788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.866747 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57544fb856-z82lr"] Jul 28 20:02:00.869024 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.869005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:00.890546 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.890516 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57544fb856-z82lr"] Jul 28 20:02:00.982403 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.982366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-console-config\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:00.982403 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.982405 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96th\" (UniqueName: \"kubernetes.io/projected/a8d4db76-00e0-43b4-8f74-0816db219a59-kube-api-access-t96th\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:00.982876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.982422 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-trusted-ca-bundle\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:00.982876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.982522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-oauth-serving-cert\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:00.982876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.982592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-serving-cert\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:00.982876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.982624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-service-ca\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:00.982876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:00.982684 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-oauth-config\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.083764 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.083729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-serving-cert\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.083764 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.083766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-service-ca\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.084010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.083883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-oauth-config\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.084010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.083936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-console-config\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.084086 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.084062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t96th\" (UniqueName: \"kubernetes.io/projected/a8d4db76-00e0-43b4-8f74-0816db219a59-kube-api-access-t96th\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.084120 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.084098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-trusted-ca-bundle\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.084235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.084212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-oauth-serving-cert\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.085035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.085009 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-service-ca\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.085145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.085009 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-oauth-serving-cert\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.085145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.085085 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-console-config\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.085145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.085125 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-trusted-ca-bundle\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.086449 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.086417 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-oauth-config\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.086650 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.086632 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-serving-cert\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.093534 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.093512 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96th\" (UniqueName: \"kubernetes.io/projected/a8d4db76-00e0-43b4-8f74-0816db219a59-kube-api-access-t96th\") pod \"console-57544fb856-z82lr\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.178371 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.178263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:01.314427 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.314396 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57544fb856-z82lr"] Jul 28 20:02:01.316696 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:02:01.316662 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d4db76_00e0_43b4_8f74_0816db219a59.slice/crio-3dc5cdbdf4eb970a0f4a1bad44a35f27e3a543572112b4a40537418debf50428 WatchSource:0}: Error finding container 3dc5cdbdf4eb970a0f4a1bad44a35f27e3a543572112b4a40537418debf50428: Status 404 returned error can't find the container with id 3dc5cdbdf4eb970a0f4a1bad44a35f27e3a543572112b4a40537418debf50428 Jul 28 20:02:01.967533 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.967497 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57544fb856-z82lr" event={"ID":"a8d4db76-00e0-43b4-8f74-0816db219a59","Type":"ContainerStarted","Data":"f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf"} Jul 28 20:02:01.967706 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.967548 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57544fb856-z82lr" event={"ID":"a8d4db76-00e0-43b4-8f74-0816db219a59","Type":"ContainerStarted","Data":"3dc5cdbdf4eb970a0f4a1bad44a35f27e3a543572112b4a40537418debf50428"} Jul 28 20:02:01.990530 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:01.990461 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57544fb856-z82lr" podStartSLOduration=1.990441003 podStartE2EDuration="1.990441003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:02:01.989367346 +0000 UTC m=+149.433678363" watchObservedRunningTime="2026-07-28 20:02:01.990441003 +0000 UTC m=+149.434752021" Jul 28 20:02:11.178969 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:11.178924 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:11.179412 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:11.179010 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:11.183563 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:11.183541 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:12.011156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:12.011129 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:02:12.059830 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:12.059793 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b78d555cb-bhg5x"] Jul 28 20:02:37.079272 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.079218 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b78d555cb-bhg5x" podUID="32810a11-cfd4-4187-bf69-1c8779adbb7a" containerName="console" containerID="cri-o://73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9" gracePeriod=15 Jul 28 20:02:37.318716 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.318690 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:02:37.378983 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.378911 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-trusted-ca-bundle\") pod \"32810a11-cfd4-4187-bf69-1c8779adbb7a\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " Jul 28 20:02:37.378983 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.378975 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-service-ca\") pod \"32810a11-cfd4-4187-bf69-1c8779adbb7a\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " Jul 28 20:02:37.379188 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379000 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-oauth-serving-cert\") pod \"32810a11-cfd4-4187-bf69-1c8779adbb7a\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " Jul 28 20:02:37.379188 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379016 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-config\") pod \"32810a11-cfd4-4187-bf69-1c8779adbb7a\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " Jul 28 20:02:37.379188 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379034 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-oauth-config\") pod \"32810a11-cfd4-4187-bf69-1c8779adbb7a\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " Jul 28 20:02:37.379188 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379060 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv6kv\" (UniqueName: \"kubernetes.io/projected/32810a11-cfd4-4187-bf69-1c8779adbb7a-kube-api-access-sv6kv\") pod \"32810a11-cfd4-4187-bf69-1c8779adbb7a\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " Jul 28 20:02:37.379188 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379081 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-serving-cert\") pod \"32810a11-cfd4-4187-bf69-1c8779adbb7a\" (UID: \"32810a11-cfd4-4187-bf69-1c8779adbb7a\") " Jul 28 20:02:37.379451 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379428 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "32810a11-cfd4-4187-bf69-1c8779adbb7a" (UID: "32810a11-cfd4-4187-bf69-1c8779adbb7a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:02:37.379508 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379448 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "32810a11-cfd4-4187-bf69-1c8779adbb7a" (UID: "32810a11-cfd4-4187-bf69-1c8779adbb7a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:02:37.379562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379541 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-service-ca" (OuterVolumeSpecName: "service-ca") pod "32810a11-cfd4-4187-bf69-1c8779adbb7a" (UID: "32810a11-cfd4-4187-bf69-1c8779adbb7a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:02:37.379609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.379552 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-config" (OuterVolumeSpecName: "console-config") pod "32810a11-cfd4-4187-bf69-1c8779adbb7a" (UID: "32810a11-cfd4-4187-bf69-1c8779adbb7a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:02:37.381374 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.381344 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "32810a11-cfd4-4187-bf69-1c8779adbb7a" (UID: "32810a11-cfd4-4187-bf69-1c8779adbb7a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:02:37.381374 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.381369 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32810a11-cfd4-4187-bf69-1c8779adbb7a-kube-api-access-sv6kv" (OuterVolumeSpecName: "kube-api-access-sv6kv") pod "32810a11-cfd4-4187-bf69-1c8779adbb7a" (UID: "32810a11-cfd4-4187-bf69-1c8779adbb7a"). InnerVolumeSpecName "kube-api-access-sv6kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:02:37.381544 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.381521 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "32810a11-cfd4-4187-bf69-1c8779adbb7a" (UID: "32810a11-cfd4-4187-bf69-1c8779adbb7a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:02:37.480492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.480452 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:02:37.480492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.480488 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-oauth-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:02:37.480492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.480497 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:02:37.480492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.480508 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-oauth-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:02:37.480751 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.480516 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6kv\" (UniqueName: \"kubernetes.io/projected/32810a11-cfd4-4187-bf69-1c8779adbb7a-kube-api-access-sv6kv\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:02:37.480751 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.480525 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32810a11-cfd4-4187-bf69-1c8779adbb7a-console-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:02:37.480751 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:37.480533 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32810a11-cfd4-4187-bf69-1c8779adbb7a-trusted-ca-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:02:38.107281 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.107245 2581 generic.go:356] "Generic (PLEG): container finished" podID="32810a11-cfd4-4187-bf69-1c8779adbb7a" containerID="73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9" exitCode=0 Jul 28 20:02:38.107677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.107289 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b78d555cb-bhg5x" Jul 28 20:02:38.107677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.107335 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78d555cb-bhg5x" event={"ID":"32810a11-cfd4-4187-bf69-1c8779adbb7a","Type":"ContainerDied","Data":"73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9"} Jul 28 20:02:38.107677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.107385 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b78d555cb-bhg5x" event={"ID":"32810a11-cfd4-4187-bf69-1c8779adbb7a","Type":"ContainerDied","Data":"f48a7188d7b82ae1f7fa4c295f451ec7a3d857b49cc9ca096a14e9d50882ce2e"} Jul 28 20:02:38.107677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.107406 2581 scope.go:117] "RemoveContainer" containerID="73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9" Jul 28 20:02:38.116437 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.116422 2581 scope.go:117] "RemoveContainer" containerID="73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9" Jul 28 20:02:38.116698 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:02:38.116678 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9\": container with ID starting with 73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9 not found: ID does not exist" containerID="73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9" Jul 28 20:02:38.116791 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.116712 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9"} err="failed to get container status \"73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9\": rpc error: code = NotFound desc = could not find container \"73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9\": container with ID starting with 73c0e32b76b00acd6e07954f463815623d80e18f681e12bdb0730bbf9d80c4b9 not found: ID does not exist" Jul 28 20:02:38.127022 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.126995 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b78d555cb-bhg5x"] Jul 28 20:02:38.130550 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:38.130532 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b78d555cb-bhg5x"] Jul 28 20:02:39.150222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:02:39.150192 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32810a11-cfd4-4187-bf69-1c8779adbb7a" path="/var/lib/kubelet/pods/32810a11-cfd4-4187-bf69-1c8779adbb7a/volumes" Jul 28 20:03:07.379663 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.379634 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-777c7df6fd-kt2db"] Jul 28 20:03:07.380148 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.379928 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32810a11-cfd4-4187-bf69-1c8779adbb7a" containerName="console" Jul 28 20:03:07.380148 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.379939 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="32810a11-cfd4-4187-bf69-1c8779adbb7a" containerName="console" Jul 28 20:03:07.380148 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.380004 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="32810a11-cfd4-4187-bf69-1c8779adbb7a" containerName="console" Jul 28 20:03:07.382701 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.382682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.395948 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.395924 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777c7df6fd-kt2db"] Jul 28 20:03:07.430943 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.430907 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-oauth-serving-cert\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.431092 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.430966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-service-ca\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.431092 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.431022 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-oauth-config\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.431092 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.431044 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-config\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.431218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.431092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bq5\" (UniqueName: \"kubernetes.io/projected/681fdd0a-ba30-4750-83a9-8b7280b02b0a-kube-api-access-78bq5\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.431218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.431121 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-trusted-ca-bundle\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.431218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.431193 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-serving-cert\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532180 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-oauth-serving-cert\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-service-ca\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-oauth-config\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-config\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532284 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bq5\" (UniqueName: \"kubernetes.io/projected/681fdd0a-ba30-4750-83a9-8b7280b02b0a-kube-api-access-78bq5\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532520 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532312 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-trusted-ca-bundle\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532520 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-serving-cert\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.532931 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.532906 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-oauth-serving-cert\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.533056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.533033 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-service-ca\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.533153 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.533139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-config\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.533193 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.533141 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-trusted-ca-bundle\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.535188 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.535172 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-oauth-config\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.535424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.535404 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-serving-cert\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.539588 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.539562 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bq5\" (UniqueName: \"kubernetes.io/projected/681fdd0a-ba30-4750-83a9-8b7280b02b0a-kube-api-access-78bq5\") pod \"console-777c7df6fd-kt2db\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.692756 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.692678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:07.822075 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:07.822044 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-777c7df6fd-kt2db"] Jul 28 20:03:07.825427 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:03:07.825391 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681fdd0a_ba30_4750_83a9_8b7280b02b0a.slice/crio-0cfbeb8267abed2985895d89a3252f05e22be79debc45c12185abb3e84792c7f WatchSource:0}: Error finding container 0cfbeb8267abed2985895d89a3252f05e22be79debc45c12185abb3e84792c7f: Status 404 returned error can't find the container with id 0cfbeb8267abed2985895d89a3252f05e22be79debc45c12185abb3e84792c7f Jul 28 20:03:08.233778 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:08.233746 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777c7df6fd-kt2db" event={"ID":"681fdd0a-ba30-4750-83a9-8b7280b02b0a","Type":"ContainerStarted","Data":"df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3"} Jul 28 20:03:08.233937 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:08.233792 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777c7df6fd-kt2db" event={"ID":"681fdd0a-ba30-4750-83a9-8b7280b02b0a","Type":"ContainerStarted","Data":"0cfbeb8267abed2985895d89a3252f05e22be79debc45c12185abb3e84792c7f"} Jul 28 20:03:08.249236 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:08.249188 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-777c7df6fd-kt2db" podStartSLOduration=1.249173241 podStartE2EDuration="1.249173241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:03:08.248704288 +0000 UTC m=+215.693015304" watchObservedRunningTime="2026-07-28 20:03:08.249173241 +0000 UTC m=+215.693484259" Jul 28 20:03:17.692778 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:17.692742 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:17.693382 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:17.692835 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:17.697514 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:17.697487 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:18.275143 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:18.275116 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:03:18.320973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:18.320936 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57544fb856-z82lr"] Jul 28 20:03:43.342021 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.341946 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57544fb856-z82lr" podUID="a8d4db76-00e0-43b4-8f74-0816db219a59" containerName="console" containerID="cri-o://f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf" gracePeriod=15 Jul 28 20:03:43.583673 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.583649 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:03:43.737253 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737227 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-console-config\") pod \"a8d4db76-00e0-43b4-8f74-0816db219a59\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " Jul 28 20:03:43.737429 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737259 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-service-ca\") pod \"a8d4db76-00e0-43b4-8f74-0816db219a59\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " Jul 28 20:03:43.737429 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737285 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-oauth-serving-cert\") pod \"a8d4db76-00e0-43b4-8f74-0816db219a59\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " Jul 28 20:03:43.737429 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737313 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-trusted-ca-bundle\") pod \"a8d4db76-00e0-43b4-8f74-0816db219a59\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " Jul 28 20:03:43.737429 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737395 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-serving-cert\") pod \"a8d4db76-00e0-43b4-8f74-0816db219a59\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " Jul 28 20:03:43.737633 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737443 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-oauth-config\") pod \"a8d4db76-00e0-43b4-8f74-0816db219a59\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " Jul 28 20:03:43.737633 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737500 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t96th\" (UniqueName: \"kubernetes.io/projected/a8d4db76-00e0-43b4-8f74-0816db219a59-kube-api-access-t96th\") pod \"a8d4db76-00e0-43b4-8f74-0816db219a59\" (UID: \"a8d4db76-00e0-43b4-8f74-0816db219a59\") " Jul 28 20:03:43.737739 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737666 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-console-config" (OuterVolumeSpecName: "console-config") pod "a8d4db76-00e0-43b4-8f74-0816db219a59" (UID: "a8d4db76-00e0-43b4-8f74-0816db219a59"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:03:43.737739 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737697 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-service-ca" (OuterVolumeSpecName: "service-ca") pod "a8d4db76-00e0-43b4-8f74-0816db219a59" (UID: "a8d4db76-00e0-43b4-8f74-0816db219a59"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:03:43.737834 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737735 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a8d4db76-00e0-43b4-8f74-0816db219a59" (UID: "a8d4db76-00e0-43b4-8f74-0816db219a59"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:03:43.737834 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.737743 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-console-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:03:43.738128 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.738097 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a8d4db76-00e0-43b4-8f74-0816db219a59" (UID: "a8d4db76-00e0-43b4-8f74-0816db219a59"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:03:43.739596 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.739573 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a8d4db76-00e0-43b4-8f74-0816db219a59" (UID: "a8d4db76-00e0-43b4-8f74-0816db219a59"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:03:43.739680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.739604 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a8d4db76-00e0-43b4-8f74-0816db219a59" (UID: "a8d4db76-00e0-43b4-8f74-0816db219a59"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:03:43.739736 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.739692 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d4db76-00e0-43b4-8f74-0816db219a59-kube-api-access-t96th" (OuterVolumeSpecName: "kube-api-access-t96th") pod "a8d4db76-00e0-43b4-8f74-0816db219a59" (UID: "a8d4db76-00e0-43b4-8f74-0816db219a59"). InnerVolumeSpecName "kube-api-access-t96th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:03:43.838875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.838830 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-oauth-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:03:43.838875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.838869 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t96th\" (UniqueName: \"kubernetes.io/projected/a8d4db76-00e0-43b4-8f74-0816db219a59-kube-api-access-t96th\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:03:43.838875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.838881 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:03:43.838875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.838890 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-oauth-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:03:43.839137 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.838900 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8d4db76-00e0-43b4-8f74-0816db219a59-trusted-ca-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:03:43.839137 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:43.838909 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8d4db76-00e0-43b4-8f74-0816db219a59-console-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:03:44.372335 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.372290 2581 generic.go:356] "Generic (PLEG): container finished" podID="a8d4db76-00e0-43b4-8f74-0816db219a59" containerID="f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf" exitCode=0 Jul 28 20:03:44.372772 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.372352 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57544fb856-z82lr" Jul 28 20:03:44.372772 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.372364 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57544fb856-z82lr" event={"ID":"a8d4db76-00e0-43b4-8f74-0816db219a59","Type":"ContainerDied","Data":"f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf"} Jul 28 20:03:44.372772 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.372413 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57544fb856-z82lr" event={"ID":"a8d4db76-00e0-43b4-8f74-0816db219a59","Type":"ContainerDied","Data":"3dc5cdbdf4eb970a0f4a1bad44a35f27e3a543572112b4a40537418debf50428"} Jul 28 20:03:44.372772 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.372440 2581 scope.go:117] "RemoveContainer" containerID="f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf" Jul 28 20:03:44.380884 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.380867 2581 scope.go:117] "RemoveContainer" containerID="f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf" Jul 28 20:03:44.381151 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:03:44.381132 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf\": container with ID starting with f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf not found: ID does not exist" containerID="f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf" Jul 28 20:03:44.381209 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.381163 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf"} err="failed to get container status \"f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf\": rpc error: code = NotFound desc = could not find container \"f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf\": container with ID starting with f03fb49dc16f37fb16c0b3935725345a71c414bc3389c1ee22c9341c2e6cf4bf not found: ID does not exist" Jul 28 20:03:44.391386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.391358 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57544fb856-z82lr"] Jul 28 20:03:44.394627 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:44.394608 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57544fb856-z82lr"] Jul 28 20:03:45.150667 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:03:45.150636 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d4db76-00e0-43b4-8f74-0816db219a59" path="/var/lib/kubelet/pods/a8d4db76-00e0-43b4-8f74-0816db219a59/volumes" Jul 28 20:04:08.959877 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.959797 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb"] Jul 28 20:04:08.960351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.960175 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d4db76-00e0-43b4-8f74-0816db219a59" containerName="console" Jul 28 20:04:08.960351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.960188 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d4db76-00e0-43b4-8f74-0816db219a59" containerName="console" Jul 28 20:04:08.960351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.960251 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d4db76-00e0-43b4-8f74-0816db219a59" containerName="console" Jul 28 20:04:08.963256 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.963241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:08.965460 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.965432 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 20:04:08.965460 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.965437 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t48d5\"" Jul 28 20:04:08.966008 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.965991 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 20:04:08.970904 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:08.970883 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb"] Jul 28 20:04:09.042872 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.042836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.043071 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.042878 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.043071 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.042999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tvt\" (UniqueName: \"kubernetes.io/projected/11d69a2b-b71d-44d6-a140-326f073d6df9-kube-api-access-r7tvt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.143414 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.143377 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tvt\" (UniqueName: \"kubernetes.io/projected/11d69a2b-b71d-44d6-a140-326f073d6df9-kube-api-access-r7tvt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.143618 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.143429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.143618 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.143538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.143850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.143827 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.143850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.143844 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.151524 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.151493 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tvt\" (UniqueName: \"kubernetes.io/projected/11d69a2b-b71d-44d6-a140-326f073d6df9-kube-api-access-r7tvt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.273068 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.273033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:09.402885 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.402858 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb"] Jul 28 20:04:09.405375 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:04:09.405349 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d69a2b_b71d_44d6_a140_326f073d6df9.slice/crio-2b5b3d0f44e4983e8c84a1faee4059b51fd5073822f43d180d90c0da81f8df1a WatchSource:0}: Error finding container 2b5b3d0f44e4983e8c84a1faee4059b51fd5073822f43d180d90c0da81f8df1a: Status 404 returned error can't find the container with id 2b5b3d0f44e4983e8c84a1faee4059b51fd5073822f43d180d90c0da81f8df1a Jul 28 20:04:09.467792 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:09.467759 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" event={"ID":"11d69a2b-b71d-44d6-a140-326f073d6df9","Type":"ContainerStarted","Data":"2b5b3d0f44e4983e8c84a1faee4059b51fd5073822f43d180d90c0da81f8df1a"} Jul 28 20:04:18.510378 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:18.510342 2581 generic.go:356] "Generic (PLEG): container finished" podID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerID="b365b21db1885ac3e83d39c240a0ba75a3e073b74f8364f9bc682e19774cf820" exitCode=0 Jul 28 20:04:18.510814 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:18.510400 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" event={"ID":"11d69a2b-b71d-44d6-a140-326f073d6df9","Type":"ContainerDied","Data":"b365b21db1885ac3e83d39c240a0ba75a3e073b74f8364f9bc682e19774cf820"} Jul 28 20:04:20.521461 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:20.521422 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" event={"ID":"11d69a2b-b71d-44d6-a140-326f073d6df9","Type":"ContainerStarted","Data":"bab32d5ea46fe7e5e2310f6ba677d2d504546683710083bf7b47adac374bc3bf"} Jul 28 20:04:21.527732 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:21.527697 2581 generic.go:356] "Generic (PLEG): container finished" podID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerID="bab32d5ea46fe7e5e2310f6ba677d2d504546683710083bf7b47adac374bc3bf" exitCode=0 Jul 28 20:04:21.528134 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:21.527777 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" event={"ID":"11d69a2b-b71d-44d6-a140-326f073d6df9","Type":"ContainerDied","Data":"bab32d5ea46fe7e5e2310f6ba677d2d504546683710083bf7b47adac374bc3bf"} Jul 28 20:04:28.559405 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:28.559368 2581 generic.go:356] "Generic (PLEG): container finished" podID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerID="2b011ec0de9d1b4a93053e68e09ccc93231d4df77780808ec22c5c429f2634d9" exitCode=0 Jul 28 20:04:28.559797 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:28.559454 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" event={"ID":"11d69a2b-b71d-44d6-a140-326f073d6df9","Type":"ContainerDied","Data":"2b011ec0de9d1b4a93053e68e09ccc93231d4df77780808ec22c5c429f2634d9"} Jul 28 20:04:29.687874 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.687844 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:29.830903 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.830804 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-util\") pod \"11d69a2b-b71d-44d6-a140-326f073d6df9\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " Jul 28 20:04:29.830903 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.830842 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-bundle\") pod \"11d69a2b-b71d-44d6-a140-326f073d6df9\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " Jul 28 20:04:29.830903 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.830882 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7tvt\" (UniqueName: \"kubernetes.io/projected/11d69a2b-b71d-44d6-a140-326f073d6df9-kube-api-access-r7tvt\") pod \"11d69a2b-b71d-44d6-a140-326f073d6df9\" (UID: \"11d69a2b-b71d-44d6-a140-326f073d6df9\") " Jul 28 20:04:29.831559 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.831532 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-bundle" (OuterVolumeSpecName: "bundle") pod "11d69a2b-b71d-44d6-a140-326f073d6df9" (UID: "11d69a2b-b71d-44d6-a140-326f073d6df9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:04:29.833039 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.833019 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d69a2b-b71d-44d6-a140-326f073d6df9-kube-api-access-r7tvt" (OuterVolumeSpecName: "kube-api-access-r7tvt") pod "11d69a2b-b71d-44d6-a140-326f073d6df9" (UID: "11d69a2b-b71d-44d6-a140-326f073d6df9"). InnerVolumeSpecName "kube-api-access-r7tvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:04:29.836556 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.836532 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-util" (OuterVolumeSpecName: "util") pod "11d69a2b-b71d-44d6-a140-326f073d6df9" (UID: "11d69a2b-b71d-44d6-a140-326f073d6df9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:04:29.931956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.931902 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-util\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:04:29.931956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.931948 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11d69a2b-b71d-44d6-a140-326f073d6df9-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:04:29.931956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:29.931960 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7tvt\" (UniqueName: \"kubernetes.io/projected/11d69a2b-b71d-44d6-a140-326f073d6df9-kube-api-access-r7tvt\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:04:30.569752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:30.569721 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" Jul 28 20:04:30.569953 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:30.569720 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540sc6mb" event={"ID":"11d69a2b-b71d-44d6-a140-326f073d6df9","Type":"ContainerDied","Data":"2b5b3d0f44e4983e8c84a1faee4059b51fd5073822f43d180d90c0da81f8df1a"} Jul 28 20:04:30.569953 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:30.569837 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b5b3d0f44e4983e8c84a1faee4059b51fd5073822f43d180d90c0da81f8df1a" Jul 28 20:04:32.957282 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:32.957258 2581 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 20:04:37.071668 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.071625 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg"] Jul 28 20:04:37.073228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.071926 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerName="pull" Jul 28 20:04:37.073228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.071938 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerName="pull" Jul 28 20:04:37.073228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.071953 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerName="extract" Jul 28 20:04:37.073228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.071959 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerName="extract" Jul 28 20:04:37.073228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.071977 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerName="util" Jul 28 20:04:37.073228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.071983 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerName="util" Jul 28 20:04:37.073228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.072035 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d69a2b-b71d-44d6-a140-326f073d6df9" containerName="extract" Jul 28 20:04:37.075769 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.075749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.080210 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.080184 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4h2g6\"" Jul 28 20:04:37.080422 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.080402 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 20:04:37.084473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.084176 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 20:04:37.103062 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.103032 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg"] Jul 28 20:04:37.194474 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.194414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6118b985-96a8-433a-bbac-86e8577709db-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zptcg\" (UID: \"6118b985-96a8-433a-bbac-86e8577709db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.194474 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.194469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9qx\" (UniqueName: \"kubernetes.io/projected/6118b985-96a8-433a-bbac-86e8577709db-kube-api-access-db9qx\") pod \"cert-manager-operator-controller-manager-55c444bf86-zptcg\" (UID: \"6118b985-96a8-433a-bbac-86e8577709db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.295054 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.295005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6118b985-96a8-433a-bbac-86e8577709db-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zptcg\" (UID: \"6118b985-96a8-433a-bbac-86e8577709db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.295206 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.295060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9qx\" (UniqueName: \"kubernetes.io/projected/6118b985-96a8-433a-bbac-86e8577709db-kube-api-access-db9qx\") pod \"cert-manager-operator-controller-manager-55c444bf86-zptcg\" (UID: \"6118b985-96a8-433a-bbac-86e8577709db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.295517 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.295497 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6118b985-96a8-433a-bbac-86e8577709db-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zptcg\" (UID: \"6118b985-96a8-433a-bbac-86e8577709db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.315502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.315464 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9qx\" (UniqueName: \"kubernetes.io/projected/6118b985-96a8-433a-bbac-86e8577709db-kube-api-access-db9qx\") pod \"cert-manager-operator-controller-manager-55c444bf86-zptcg\" (UID: \"6118b985-96a8-433a-bbac-86e8577709db\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.387191 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.387084 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" Jul 28 20:04:37.526535 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.526503 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg"] Jul 28 20:04:37.529415 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:04:37.529382 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6118b985_96a8_433a_bbac_86e8577709db.slice/crio-ebcd5064866e50efea00ca75d017e087a27a65cfb3b17491714fda462d72cfaf WatchSource:0}: Error finding container ebcd5064866e50efea00ca75d017e087a27a65cfb3b17491714fda462d72cfaf: Status 404 returned error can't find the container with id ebcd5064866e50efea00ca75d017e087a27a65cfb3b17491714fda462d72cfaf Jul 28 20:04:37.531879 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.531860 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:04:37.599218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:37.599184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" event={"ID":"6118b985-96a8-433a-bbac-86e8577709db","Type":"ContainerStarted","Data":"ebcd5064866e50efea00ca75d017e087a27a65cfb3b17491714fda462d72cfaf"} Jul 28 20:04:39.609742 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:39.609707 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" event={"ID":"6118b985-96a8-433a-bbac-86e8577709db","Type":"ContainerStarted","Data":"4f0b9f94ba77bcca1273fab212e92c2d948dc9d962c279d98c773309b679406e"} Jul 28 20:04:41.000122 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.000063 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zptcg" podStartSLOduration=2.3994159489999998 podStartE2EDuration="4.000042572s" totalImagesPullingTime="1.600626623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:04:39.643265276 +0000 UTC m=+307.087576291" watchObservedRunningTime="2026-07-28 20:04:41.000042572 +0000 UTC m=+308.444353597" Jul 28 20:04:41.001203 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.001178 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f6wrw"] Jul 28 20:04:41.003697 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.003678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.005951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.005927 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 20:04:41.006485 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.006458 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 20:04:41.006598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.006552 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zqkt5\"" Jul 28 20:04:41.014260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.014241 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f6wrw"] Jul 28 20:04:41.024440 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.024403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a2843a5-b20f-4ffd-8244-ab308bb97bd3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f6wrw\" (UID: \"0a2843a5-b20f-4ffd-8244-ab308bb97bd3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.024578 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.024523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzp4f\" (UniqueName: \"kubernetes.io/projected/0a2843a5-b20f-4ffd-8244-ab308bb97bd3-kube-api-access-fzp4f\") pod \"cert-manager-webhook-6486ffd997-f6wrw\" (UID: \"0a2843a5-b20f-4ffd-8244-ab308bb97bd3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.050010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.049977 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr"] Jul 28 20:04:41.054768 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.054743 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.056947 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.056919 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 20:04:41.057058 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.056929 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t48d5\"" Jul 28 20:04:41.057058 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.057008 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 20:04:41.061830 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.061804 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr"] Jul 28 20:04:41.125711 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.125669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzp4f\" (UniqueName: \"kubernetes.io/projected/0a2843a5-b20f-4ffd-8244-ab308bb97bd3-kube-api-access-fzp4f\") pod \"cert-manager-webhook-6486ffd997-f6wrw\" (UID: \"0a2843a5-b20f-4ffd-8244-ab308bb97bd3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.125913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.125727 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.125913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.125783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.125913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.125814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a2843a5-b20f-4ffd-8244-ab308bb97bd3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f6wrw\" (UID: \"0a2843a5-b20f-4ffd-8244-ab308bb97bd3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.125913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.125857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvbc\" (UniqueName: \"kubernetes.io/projected/f1e61099-6bdc-47f8-87c9-be48140af5e0-kube-api-access-sdvbc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.146756 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.146725 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a2843a5-b20f-4ffd-8244-ab308bb97bd3-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f6wrw\" (UID: \"0a2843a5-b20f-4ffd-8244-ab308bb97bd3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.146908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.146785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzp4f\" (UniqueName: \"kubernetes.io/projected/0a2843a5-b20f-4ffd-8244-ab308bb97bd3-kube-api-access-fzp4f\") pod \"cert-manager-webhook-6486ffd997-f6wrw\" (UID: \"0a2843a5-b20f-4ffd-8244-ab308bb97bd3\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.226607 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.226575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvbc\" (UniqueName: \"kubernetes.io/projected/f1e61099-6bdc-47f8-87c9-be48140af5e0-kube-api-access-sdvbc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.226769 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.226643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.226769 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.226729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.227055 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.227034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.227098 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.227062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.236262 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.236236 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvbc\" (UniqueName: \"kubernetes.io/projected/f1e61099-6bdc-47f8-87c9-be48140af5e0-kube-api-access-sdvbc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.322898 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.322807 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:41.366513 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.366480 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:41.475285 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.475255 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f6wrw"] Jul 28 20:04:41.477152 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:04:41.477126 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a2843a5_b20f_4ffd_8244_ab308bb97bd3.slice/crio-93bb7a126bbb86d6d154bc5310a1c6310fef6abb5413deb9b84571fd445e4b0e WatchSource:0}: Error finding container 93bb7a126bbb86d6d154bc5310a1c6310fef6abb5413deb9b84571fd445e4b0e: Status 404 returned error can't find the container with id 93bb7a126bbb86d6d154bc5310a1c6310fef6abb5413deb9b84571fd445e4b0e Jul 28 20:04:41.516484 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.516458 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr"] Jul 28 20:04:41.518705 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:04:41.518683 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e61099_6bdc_47f8_87c9_be48140af5e0.slice/crio-fc4b89fb88080a328c68ccd2bec4f56cb6b3cf5d0faea0b2ee5c281a6b06de69 WatchSource:0}: Error finding container fc4b89fb88080a328c68ccd2bec4f56cb6b3cf5d0faea0b2ee5c281a6b06de69: Status 404 returned error can't find the container with id fc4b89fb88080a328c68ccd2bec4f56cb6b3cf5d0faea0b2ee5c281a6b06de69 Jul 28 20:04:41.620172 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.620146 2581 generic.go:356] "Generic (PLEG): container finished" podID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerID="54f321f33feff574aa589acc11f67a86e393e768f7399fa61d8b2fd3e0ea59a0" exitCode=0 Jul 28 20:04:41.620300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.620217 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" event={"ID":"f1e61099-6bdc-47f8-87c9-be48140af5e0","Type":"ContainerDied","Data":"54f321f33feff574aa589acc11f67a86e393e768f7399fa61d8b2fd3e0ea59a0"} Jul 28 20:04:41.620300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.620261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" event={"ID":"f1e61099-6bdc-47f8-87c9-be48140af5e0","Type":"ContainerStarted","Data":"fc4b89fb88080a328c68ccd2bec4f56cb6b3cf5d0faea0b2ee5c281a6b06de69"} Jul 28 20:04:41.621461 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:41.621429 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" event={"ID":"0a2843a5-b20f-4ffd-8244-ab308bb97bd3","Type":"ContainerStarted","Data":"93bb7a126bbb86d6d154bc5310a1c6310fef6abb5413deb9b84571fd445e4b0e"} Jul 28 20:04:44.639473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:44.639441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" event={"ID":"0a2843a5-b20f-4ffd-8244-ab308bb97bd3","Type":"ContainerStarted","Data":"edeaa3d42218e8dc26c55d85437379f6baaef2d012e1bfc6ede7f980d16bade2"} Jul 28 20:04:44.639896 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:44.639499 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:44.659870 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:44.659822 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" podStartSLOduration=1.723526055 podStartE2EDuration="4.65980552s" totalImagesPullingTime="2.936279465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:04:44.658478931 +0000 UTC m=+312.102789946" watchObservedRunningTime="2026-07-28 20:04:44.65980552 +0000 UTC m=+312.104116536" Jul 28 20:04:45.038133 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.038102 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w444f"] Jul 28 20:04:45.040376 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.040362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.042187 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.042169 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kvzsh\"" Jul 28 20:04:45.050679 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.050657 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w444f"] Jul 28 20:04:45.052064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.052035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf259280-0f6a-46a9-97b6-d1c793a6bdf9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w444f\" (UID: \"cf259280-0f6a-46a9-97b6-d1c793a6bdf9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.052160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.052071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklb9\" (UniqueName: \"kubernetes.io/projected/cf259280-0f6a-46a9-97b6-d1c793a6bdf9-kube-api-access-jklb9\") pod \"cert-manager-cainjector-559cdd8c87-w444f\" (UID: \"cf259280-0f6a-46a9-97b6-d1c793a6bdf9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.157340 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.154028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf259280-0f6a-46a9-97b6-d1c793a6bdf9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w444f\" (UID: \"cf259280-0f6a-46a9-97b6-d1c793a6bdf9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.157340 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.154095 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklb9\" (UniqueName: \"kubernetes.io/projected/cf259280-0f6a-46a9-97b6-d1c793a6bdf9-kube-api-access-jklb9\") pod \"cert-manager-cainjector-559cdd8c87-w444f\" (UID: \"cf259280-0f6a-46a9-97b6-d1c793a6bdf9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.163031 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.163006 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf259280-0f6a-46a9-97b6-d1c793a6bdf9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-w444f\" (UID: \"cf259280-0f6a-46a9-97b6-d1c793a6bdf9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.163164 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.163146 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklb9\" (UniqueName: \"kubernetes.io/projected/cf259280-0f6a-46a9-97b6-d1c793a6bdf9-kube-api-access-jklb9\") pod \"cert-manager-cainjector-559cdd8c87-w444f\" (UID: \"cf259280-0f6a-46a9-97b6-d1c793a6bdf9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.349228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.349137 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" Jul 28 20:04:45.477415 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.477314 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-w444f"] Jul 28 20:04:45.479535 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:04:45.479509 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf259280_0f6a_46a9_97b6_d1c793a6bdf9.slice/crio-d20c4b7645c7a35ce1f4ec6e494dc31d5a9a8fcb179fc7868803a61158610630 WatchSource:0}: Error finding container d20c4b7645c7a35ce1f4ec6e494dc31d5a9a8fcb179fc7868803a61158610630: Status 404 returned error can't find the container with id d20c4b7645c7a35ce1f4ec6e494dc31d5a9a8fcb179fc7868803a61158610630 Jul 28 20:04:45.645675 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.645574 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" event={"ID":"cf259280-0f6a-46a9-97b6-d1c793a6bdf9","Type":"ContainerStarted","Data":"11576448c6ce64916c27f9091ca3d007c8ac08a10fcf743842cda853cd01e8d9"} Jul 28 20:04:45.645675 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.645622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" event={"ID":"cf259280-0f6a-46a9-97b6-d1c793a6bdf9","Type":"ContainerStarted","Data":"d20c4b7645c7a35ce1f4ec6e494dc31d5a9a8fcb179fc7868803a61158610630"} Jul 28 20:04:45.661055 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:45.661000 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-w444f" podStartSLOduration=0.660979586 podStartE2EDuration="660.979586ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:04:45.659357033 +0000 UTC m=+313.103668051" watchObservedRunningTime="2026-07-28 20:04:45.660979586 +0000 UTC m=+313.105290673" Jul 28 20:04:50.289385 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.289349 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tc5gv"] Jul 28 20:04:50.291783 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.291764 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.293714 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.293698 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gdrxf\"" Jul 28 20:04:50.301606 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.301586 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tc5gv"] Jul 28 20:04:50.401459 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.401421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwf4g\" (UniqueName: \"kubernetes.io/projected/072178d7-4957-4b21-9362-39fdca96027a-kube-api-access-bwf4g\") pod \"cert-manager-55b4d954d5-tc5gv\" (UID: \"072178d7-4957-4b21-9362-39fdca96027a\") " pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.401459 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.401469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/072178d7-4957-4b21-9362-39fdca96027a-bound-sa-token\") pod \"cert-manager-55b4d954d5-tc5gv\" (UID: \"072178d7-4957-4b21-9362-39fdca96027a\") " pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.502645 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.502603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/072178d7-4957-4b21-9362-39fdca96027a-bound-sa-token\") pod \"cert-manager-55b4d954d5-tc5gv\" (UID: \"072178d7-4957-4b21-9362-39fdca96027a\") " pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.502797 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.502701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwf4g\" (UniqueName: \"kubernetes.io/projected/072178d7-4957-4b21-9362-39fdca96027a-kube-api-access-bwf4g\") pod \"cert-manager-55b4d954d5-tc5gv\" (UID: \"072178d7-4957-4b21-9362-39fdca96027a\") " pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.510543 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.510511 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/072178d7-4957-4b21-9362-39fdca96027a-bound-sa-token\") pod \"cert-manager-55b4d954d5-tc5gv\" (UID: \"072178d7-4957-4b21-9362-39fdca96027a\") " pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.510662 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.510576 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwf4g\" (UniqueName: \"kubernetes.io/projected/072178d7-4957-4b21-9362-39fdca96027a-kube-api-access-bwf4g\") pod \"cert-manager-55b4d954d5-tc5gv\" (UID: \"072178d7-4957-4b21-9362-39fdca96027a\") " pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.601657 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.601567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tc5gv" Jul 28 20:04:50.648868 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.648838 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f6wrw" Jul 28 20:04:50.735452 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:50.735427 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tc5gv"] Jul 28 20:04:50.737775 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:04:50.737752 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod072178d7_4957_4b21_9362_39fdca96027a.slice/crio-5dacc961302293d4c194888d4901cd53c5086c0aed2e11ec6742c28ec539c4cb WatchSource:0}: Error finding container 5dacc961302293d4c194888d4901cd53c5086c0aed2e11ec6742c28ec539c4cb: Status 404 returned error can't find the container with id 5dacc961302293d4c194888d4901cd53c5086c0aed2e11ec6742c28ec539c4cb Jul 28 20:04:51.675747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:51.675716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tc5gv" event={"ID":"072178d7-4957-4b21-9362-39fdca96027a","Type":"ContainerStarted","Data":"fa39f5b2a5a975006c642dc29d1ca3bb4c92093094670ea6a7b66e498e30b63c"} Jul 28 20:04:51.676128 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:51.675754 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tc5gv" event={"ID":"072178d7-4957-4b21-9362-39fdca96027a","Type":"ContainerStarted","Data":"5dacc961302293d4c194888d4901cd53c5086c0aed2e11ec6742c28ec539c4cb"} Jul 28 20:04:51.691535 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:51.691485 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tc5gv" podStartSLOduration=1.6914711059999998 podStartE2EDuration="1.691471106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:04:51.690285911 +0000 UTC m=+319.134596925" watchObservedRunningTime="2026-07-28 20:04:51.691471106 +0000 UTC m=+319.135782121" Jul 28 20:04:55.697844 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:55.697758 2581 generic.go:356] "Generic (PLEG): container finished" podID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerID="b67effe566d54bfbd264b672b3050136b1676e340c4340a4be306d06c2cf5320" exitCode=0 Jul 28 20:04:55.697844 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:55.697809 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" event={"ID":"f1e61099-6bdc-47f8-87c9-be48140af5e0","Type":"ContainerDied","Data":"b67effe566d54bfbd264b672b3050136b1676e340c4340a4be306d06c2cf5320"} Jul 28 20:04:56.707288 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:56.707248 2581 generic.go:356] "Generic (PLEG): container finished" podID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerID="98f6ff2fe030c8cced43dc0de714d44241d5cefc2dfad22587341f607af83a8d" exitCode=0 Jul 28 20:04:56.707672 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:56.707335 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" event={"ID":"f1e61099-6bdc-47f8-87c9-be48140af5e0","Type":"ContainerDied","Data":"98f6ff2fe030c8cced43dc0de714d44241d5cefc2dfad22587341f607af83a8d"} Jul 28 20:04:57.844741 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:57.844715 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:57.968478 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:57.968382 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdvbc\" (UniqueName: \"kubernetes.io/projected/f1e61099-6bdc-47f8-87c9-be48140af5e0-kube-api-access-sdvbc\") pod \"f1e61099-6bdc-47f8-87c9-be48140af5e0\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " Jul 28 20:04:57.968478 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:57.968442 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-bundle\") pod \"f1e61099-6bdc-47f8-87c9-be48140af5e0\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " Jul 28 20:04:57.968725 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:57.968493 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-util\") pod \"f1e61099-6bdc-47f8-87c9-be48140af5e0\" (UID: \"f1e61099-6bdc-47f8-87c9-be48140af5e0\") " Jul 28 20:04:57.968931 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:57.968906 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-bundle" (OuterVolumeSpecName: "bundle") pod "f1e61099-6bdc-47f8-87c9-be48140af5e0" (UID: "f1e61099-6bdc-47f8-87c9-be48140af5e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:04:57.970576 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:57.970541 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e61099-6bdc-47f8-87c9-be48140af5e0-kube-api-access-sdvbc" (OuterVolumeSpecName: "kube-api-access-sdvbc") pod "f1e61099-6bdc-47f8-87c9-be48140af5e0" (UID: "f1e61099-6bdc-47f8-87c9-be48140af5e0"). InnerVolumeSpecName "kube-api-access-sdvbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:04:57.972931 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:57.972910 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-util" (OuterVolumeSpecName: "util") pod "f1e61099-6bdc-47f8-87c9-be48140af5e0" (UID: "f1e61099-6bdc-47f8-87c9-be48140af5e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:04:58.069805 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:58.069766 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-util\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:04:58.069805 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:58.069799 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvbc\" (UniqueName: \"kubernetes.io/projected/f1e61099-6bdc-47f8-87c9-be48140af5e0-kube-api-access-sdvbc\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:04:58.069805 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:58.069809 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f1e61099-6bdc-47f8-87c9-be48140af5e0-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:04:58.719084 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:58.719061 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" Jul 28 20:04:58.719258 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:58.719056 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvwzwr" event={"ID":"f1e61099-6bdc-47f8-87c9-be48140af5e0","Type":"ContainerDied","Data":"fc4b89fb88080a328c68ccd2bec4f56cb6b3cf5d0faea0b2ee5c281a6b06de69"} Jul 28 20:04:58.719258 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:04:58.719187 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4b89fb88080a328c68ccd2bec4f56cb6b3cf5d0faea0b2ee5c281a6b06de69" Jul 28 20:05:04.246989 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.246953 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm"] Jul 28 20:05:04.247399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.247270 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerName="pull" Jul 28 20:05:04.247399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.247281 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerName="pull" Jul 28 20:05:04.247399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.247291 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerName="extract" Jul 28 20:05:04.247399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.247297 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerName="extract" Jul 28 20:05:04.247399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.247336 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerName="util" Jul 28 20:05:04.247399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.247342 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerName="util" Jul 28 20:05:04.247399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.247392 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e61099-6bdc-47f8-87c9-be48140af5e0" containerName="extract" Jul 28 20:05:04.249593 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.249577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.251602 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.251583 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 20:05:04.251707 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.251649 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 20:05:04.252180 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.252162 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4tplw\"" Jul 28 20:05:04.259396 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.259373 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm"] Jul 28 20:05:04.328342 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.328300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d2fa7c-e100-4e08-a5ba-c251999de893-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmhmm\" (UID: \"61d2fa7c-e100-4e08-a5ba-c251999de893\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.328536 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.328433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzlf6\" (UniqueName: \"kubernetes.io/projected/61d2fa7c-e100-4e08-a5ba-c251999de893-kube-api-access-kzlf6\") pod \"openshift-lws-operator-fd8ccff4c-fmhmm\" (UID: \"61d2fa7c-e100-4e08-a5ba-c251999de893\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.429489 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.429443 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d2fa7c-e100-4e08-a5ba-c251999de893-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmhmm\" (UID: \"61d2fa7c-e100-4e08-a5ba-c251999de893\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.429728 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.429512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzlf6\" (UniqueName: \"kubernetes.io/projected/61d2fa7c-e100-4e08-a5ba-c251999de893-kube-api-access-kzlf6\") pod \"openshift-lws-operator-fd8ccff4c-fmhmm\" (UID: \"61d2fa7c-e100-4e08-a5ba-c251999de893\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.429833 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.429812 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61d2fa7c-e100-4e08-a5ba-c251999de893-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmhmm\" (UID: \"61d2fa7c-e100-4e08-a5ba-c251999de893\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.437462 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.437437 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzlf6\" (UniqueName: \"kubernetes.io/projected/61d2fa7c-e100-4e08-a5ba-c251999de893-kube-api-access-kzlf6\") pod \"openshift-lws-operator-fd8ccff4c-fmhmm\" (UID: \"61d2fa7c-e100-4e08-a5ba-c251999de893\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.558293 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.558203 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" Jul 28 20:05:04.690050 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.690024 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm"] Jul 28 20:05:04.692251 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:05:04.692219 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d2fa7c_e100_4e08_a5ba_c251999de893.slice/crio-418557a6608dbed7ed451a3b1bde0a621512725162669fbc676583ccefb937f4 WatchSource:0}: Error finding container 418557a6608dbed7ed451a3b1bde0a621512725162669fbc676583ccefb937f4: Status 404 returned error can't find the container with id 418557a6608dbed7ed451a3b1bde0a621512725162669fbc676583ccefb937f4 Jul 28 20:05:04.747202 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:04.747165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" event={"ID":"61d2fa7c-e100-4e08-a5ba-c251999de893","Type":"ContainerStarted","Data":"418557a6608dbed7ed451a3b1bde0a621512725162669fbc676583ccefb937f4"} Jul 28 20:05:07.766801 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:07.766767 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" event={"ID":"61d2fa7c-e100-4e08-a5ba-c251999de893","Type":"ContainerStarted","Data":"cdf2acfd93bcac594a58014a17aa9b19f7431f027f148443e85be4ca39985272"} Jul 28 20:05:07.782341 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:07.782274 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmhmm" podStartSLOduration=1.7783777760000001 podStartE2EDuration="3.782254236s" totalImagesPullingTime="2.00387646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:07.780671027 +0000 UTC m=+335.224982043" watchObservedRunningTime="2026-07-28 20:05:07.782254236 +0000 UTC m=+335.226565252" Jul 28 20:05:12.084977 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.084934 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs"] Jul 28 20:05:12.087750 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.087727 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.089839 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.089816 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-t48d5\"" Jul 28 20:05:12.089972 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.089845 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 20:05:12.090409 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.090380 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 20:05:12.096644 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.096613 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs"] Jul 28 20:05:12.201049 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.201014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j528\" (UniqueName: \"kubernetes.io/projected/57d6ae1d-629d-458f-a8a9-0124be3c54c8-kube-api-access-2j528\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.201049 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.201052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.201284 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.201136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.302103 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.302061 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.302273 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.302143 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j528\" (UniqueName: \"kubernetes.io/projected/57d6ae1d-629d-458f-a8a9-0124be3c54c8-kube-api-access-2j528\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.302273 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.302171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.302579 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.302558 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.302579 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.302573 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.309762 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.309724 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j528\" (UniqueName: \"kubernetes.io/projected/57d6ae1d-629d-458f-a8a9-0124be3c54c8-kube-api-access-2j528\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.397624 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.397537 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:12.536432 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.536395 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs"] Jul 28 20:05:12.538539 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:05:12.538512 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d6ae1d_629d_458f_a8a9_0124be3c54c8.slice/crio-af94c2f2b62d98a50b23066f2c8ac6fa0888172f51f9b7b747c4379a2055ff0f WatchSource:0}: Error finding container af94c2f2b62d98a50b23066f2c8ac6fa0888172f51f9b7b747c4379a2055ff0f: Status 404 returned error can't find the container with id af94c2f2b62d98a50b23066f2c8ac6fa0888172f51f9b7b747c4379a2055ff0f Jul 28 20:05:12.796932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.796902 2581 generic.go:356] "Generic (PLEG): container finished" podID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerID="11144f17665f0b42d4299910896bed6aa0fe935df92b14e28d3b1e481452f760" exitCode=0 Jul 28 20:05:12.797109 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.796993 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" event={"ID":"57d6ae1d-629d-458f-a8a9-0124be3c54c8","Type":"ContainerDied","Data":"11144f17665f0b42d4299910896bed6aa0fe935df92b14e28d3b1e481452f760"} Jul 28 20:05:12.797109 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:12.797062 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" event={"ID":"57d6ae1d-629d-458f-a8a9-0124be3c54c8","Type":"ContainerStarted","Data":"af94c2f2b62d98a50b23066f2c8ac6fa0888172f51f9b7b747c4379a2055ff0f"} Jul 28 20:05:13.803044 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:13.803011 2581 generic.go:356] "Generic (PLEG): container finished" podID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerID="67c1a47a3f2813678e40039591b8645dff8e505c40c9462b44633e3e2143926f" exitCode=0 Jul 28 20:05:13.803565 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:13.803078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" event={"ID":"57d6ae1d-629d-458f-a8a9-0124be3c54c8","Type":"ContainerDied","Data":"67c1a47a3f2813678e40039591b8645dff8e505c40c9462b44633e3e2143926f"} Jul 28 20:05:14.809455 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:14.809422 2581 generic.go:356] "Generic (PLEG): container finished" podID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerID="12113c43e9be3bd5cd9fdea3ec4c12a8b68649f300a14b57d06e3cccb1265f09" exitCode=0 Jul 28 20:05:14.809839 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:14.809507 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" event={"ID":"57d6ae1d-629d-458f-a8a9-0124be3c54c8","Type":"ContainerDied","Data":"12113c43e9be3bd5cd9fdea3ec4c12a8b68649f300a14b57d06e3cccb1265f09"} Jul 28 20:05:15.962820 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:15.962794 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:16.139235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.139150 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-util\") pod \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " Jul 28 20:05:16.139235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.139227 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-bundle\") pod \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " Jul 28 20:05:16.139455 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.139287 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j528\" (UniqueName: \"kubernetes.io/projected/57d6ae1d-629d-458f-a8a9-0124be3c54c8-kube-api-access-2j528\") pod \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\" (UID: \"57d6ae1d-629d-458f-a8a9-0124be3c54c8\") " Jul 28 20:05:16.139887 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.139857 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-bundle" (OuterVolumeSpecName: "bundle") pod "57d6ae1d-629d-458f-a8a9-0124be3c54c8" (UID: "57d6ae1d-629d-458f-a8a9-0124be3c54c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:05:16.141216 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.141190 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d6ae1d-629d-458f-a8a9-0124be3c54c8-kube-api-access-2j528" (OuterVolumeSpecName: "kube-api-access-2j528") pod "57d6ae1d-629d-458f-a8a9-0124be3c54c8" (UID: "57d6ae1d-629d-458f-a8a9-0124be3c54c8"). InnerVolumeSpecName "kube-api-access-2j528". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:05:16.147372 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.147346 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-util" (OuterVolumeSpecName: "util") pod "57d6ae1d-629d-458f-a8a9-0124be3c54c8" (UID: "57d6ae1d-629d-458f-a8a9-0124be3c54c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:05:16.240199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.240164 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:05:16.240199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.240193 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j528\" (UniqueName: \"kubernetes.io/projected/57d6ae1d-629d-458f-a8a9-0124be3c54c8-kube-api-access-2j528\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:05:16.240199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.240203 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d6ae1d-629d-458f-a8a9-0124be3c54c8-util\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:05:16.824878 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.824843 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" event={"ID":"57d6ae1d-629d-458f-a8a9-0124be3c54c8","Type":"ContainerDied","Data":"af94c2f2b62d98a50b23066f2c8ac6fa0888172f51f9b7b747c4379a2055ff0f"} Jul 28 20:05:16.825044 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.824920 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af94c2f2b62d98a50b23066f2c8ac6fa0888172f51f9b7b747c4379a2055ff0f" Jul 28 20:05:16.825044 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:16.824852 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwrgcs" Jul 28 20:05:22.772996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.772960 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj"] Jul 28 20:05:22.773598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.773482 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerName="util" Jul 28 20:05:22.773598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.773502 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerName="util" Jul 28 20:05:22.773598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.773530 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerName="pull" Jul 28 20:05:22.773598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.773539 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerName="pull" Jul 28 20:05:22.773598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.773569 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerName="extract" Jul 28 20:05:22.773598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.773578 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerName="extract" Jul 28 20:05:22.773894 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.773655 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d6ae1d-629d-458f-a8a9-0124be3c54c8" containerName="extract" Jul 28 20:05:22.776311 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.776289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.778676 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.778658 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 20:05:22.779272 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.779255 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-48j6r\"" Jul 28 20:05:22.779361 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.779258 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 20:05:22.779361 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.779287 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 20:05:22.792735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.792706 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj"] Jul 28 20:05:22.797898 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.797877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/941dff99-8dbf-4928-91c7-3baf76d0919b-metrics-cert\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.797991 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.797922 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/941dff99-8dbf-4928-91c7-3baf76d0919b-cert\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.797991 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.797942 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh62p\" (UniqueName: \"kubernetes.io/projected/941dff99-8dbf-4928-91c7-3baf76d0919b-kube-api-access-vh62p\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.797991 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.797983 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/941dff99-8dbf-4928-91c7-3baf76d0919b-manager-config\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.898890 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.898844 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/941dff99-8dbf-4928-91c7-3baf76d0919b-metrics-cert\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.899071 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.898907 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/941dff99-8dbf-4928-91c7-3baf76d0919b-cert\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.899071 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.898931 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh62p\" (UniqueName: \"kubernetes.io/projected/941dff99-8dbf-4928-91c7-3baf76d0919b-kube-api-access-vh62p\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.899071 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.898962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/941dff99-8dbf-4928-91c7-3baf76d0919b-manager-config\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.899705 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.899681 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/941dff99-8dbf-4928-91c7-3baf76d0919b-manager-config\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.901580 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.901556 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/941dff99-8dbf-4928-91c7-3baf76d0919b-metrics-cert\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.901685 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.901564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/941dff99-8dbf-4928-91c7-3baf76d0919b-cert\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:22.913993 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:22.913967 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh62p\" (UniqueName: \"kubernetes.io/projected/941dff99-8dbf-4928-91c7-3baf76d0919b-kube-api-access-vh62p\") pod \"lws-controller-manager-789bdb4f85-6qxsj\" (UID: \"941dff99-8dbf-4928-91c7-3baf76d0919b\") " pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:23.085067 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.084974 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:23.223338 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.223300 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj"] Jul 28 20:05:23.225933 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:05:23.225906 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941dff99_8dbf_4928_91c7_3baf76d0919b.slice/crio-34639e0403542b340dae9c141e358eae030148d096e7869f9ede5d35038682fa WatchSource:0}: Error finding container 34639e0403542b340dae9c141e358eae030148d096e7869f9ede5d35038682fa: Status 404 returned error can't find the container with id 34639e0403542b340dae9c141e358eae030148d096e7869f9ede5d35038682fa Jul 28 20:05:23.560043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.560009 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw"] Jul 28 20:05:23.564184 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.564164 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.567374 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.567349 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 20:05:23.567719 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.567700 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 20:05:23.567795 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.567700 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-npzt2\"" Jul 28 20:05:23.567795 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.567732 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 20:05:23.568162 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.568141 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 20:05:23.600623 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.600595 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw"] Jul 28 20:05:23.603940 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.603918 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whm2q\" (UniqueName: \"kubernetes.io/projected/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-kube-api-access-whm2q\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.604088 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.603955 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-manifests\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.604088 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.603984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.604088 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.604039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.604088 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.604056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-tmp\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.704483 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.704452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whm2q\" (UniqueName: \"kubernetes.io/projected/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-kube-api-access-whm2q\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.704684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.704500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-manifests\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.704684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.704532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.704684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.704571 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.704684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.704589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-tmp\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.705033 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.704999 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-manifests\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.707046 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.707025 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-tmp\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.707184 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.707168 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.707242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.707226 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.712722 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.712702 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whm2q\" (UniqueName: \"kubernetes.io/projected/3ca056c4-b45b-42e3-80e7-2483d9b2f0d6-kube-api-access-whm2q\") pod \"opendatahub-operator-controller-manager-685bc94585-5tjgw\" (UID: \"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6\") " pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:23.860906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.860822 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" event={"ID":"941dff99-8dbf-4928-91c7-3baf76d0919b","Type":"ContainerStarted","Data":"34639e0403542b340dae9c141e358eae030148d096e7869f9ede5d35038682fa"} Jul 28 20:05:23.875132 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:23.875108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:24.013227 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:24.013121 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw"] Jul 28 20:05:24.016208 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:05:24.016182 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca056c4_b45b_42e3_80e7_2483d9b2f0d6.slice/crio-1f55f7358c2a53f885df306e5462e618d7cd52bcab0eb2bb85d95b2a4591e92d WatchSource:0}: Error finding container 1f55f7358c2a53f885df306e5462e618d7cd52bcab0eb2bb85d95b2a4591e92d: Status 404 returned error can't find the container with id 1f55f7358c2a53f885df306e5462e618d7cd52bcab0eb2bb85d95b2a4591e92d Jul 28 20:05:24.881368 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:24.872571 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" event={"ID":"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6","Type":"ContainerStarted","Data":"1f55f7358c2a53f885df306e5462e618d7cd52bcab0eb2bb85d95b2a4591e92d"} Jul 28 20:05:25.881930 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:25.881857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" event={"ID":"941dff99-8dbf-4928-91c7-3baf76d0919b","Type":"ContainerStarted","Data":"885d845ec2bae2e6bffa25eb7356d6e34cf5f1f05b3f19e667ba5c8e4dc9160f"} Jul 28 20:05:25.882429 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:25.882019 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:25.903335 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:25.903265 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" podStartSLOduration=1.819856711 podStartE2EDuration="3.903245688s" totalImagesPullingTime="2.083388977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:25.900768684 +0000 UTC m=+353.345079704" watchObservedRunningTime="2026-07-28 20:05:25.903245688 +0000 UTC m=+353.347556705" Jul 28 20:05:27.894060 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:27.894021 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" event={"ID":"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6","Type":"ContainerStarted","Data":"f081ba260d7a645eaf4fdd152a471b397f37d613d51a37f69d733e8715927d68"} Jul 28 20:05:28.900983 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:28.900947 2581 generic.go:356] "Generic (PLEG): container finished" podID="3ca056c4-b45b-42e3-80e7-2483d9b2f0d6" containerID="f081ba260d7a645eaf4fdd152a471b397f37d613d51a37f69d733e8715927d68" exitCode=0 Jul 28 20:05:28.901406 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:28.901033 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" event={"ID":"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6","Type":"ContainerDied","Data":"f081ba260d7a645eaf4fdd152a471b397f37d613d51a37f69d733e8715927d68"} Jul 28 20:05:29.907485 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:29.907446 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" event={"ID":"3ca056c4-b45b-42e3-80e7-2483d9b2f0d6","Type":"ContainerStarted","Data":"f7bd0a37b7972a684ace86098addc69f2c3cc068ef84317bbec2c157e5a2dc5e"} Jul 28 20:05:29.907957 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:29.907587 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:05:29.927367 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:29.927295 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" podStartSLOduration=2.376281524 podStartE2EDuration="6.927279266s" totalImagesPullingTime="3.525841374s" totalInitContainerRuntime="1.025156368s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:29.926834596 +0000 UTC m=+357.371145612" watchObservedRunningTime="2026-07-28 20:05:29.927279266 +0000 UTC m=+357.371590281" Jul 28 20:05:36.890681 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:36.890643 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-789bdb4f85-6qxsj" Jul 28 20:05:40.914294 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:05:40.914260 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-685bc94585-5tjgw" Jul 28 20:06:01.932159 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:01.932129 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d"] Jul 28 20:06:01.938567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:01.938546 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:01.940524 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:01.940491 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 20:06:01.940524 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:01.940504 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 20:06:01.940697 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:01.940586 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wbm2p\"" Jul 28 20:06:01.940697 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:01.940607 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 20:06:01.947177 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:01.947148 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d"] Jul 28 20:06:02.042880 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.042838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.042880 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.042883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.043125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.042915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.043125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.042969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.043125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.043008 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vz9\" (UniqueName: \"kubernetes.io/projected/b560ce9e-22dd-4f93-8492-ade45b36c42c-kube-api-access-j8vz9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.043125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.043032 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.043125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.043105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.043312 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.043135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.043312 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.043187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b560ce9e-22dd-4f93-8492-ade45b36c42c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b560ce9e-22dd-4f93-8492-ade45b36c42c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vz9\" (UniqueName: \"kubernetes.io/projected/b560ce9e-22dd-4f93-8492-ade45b36c42c-kube-api-access-j8vz9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.144951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.144905 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.145238 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.145133 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.145238 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.145156 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.145238 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.145210 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.145423 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.145357 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.145554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.145535 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b560ce9e-22dd-4f93-8492-ade45b36c42c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.147266 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.147239 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.147504 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.147488 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.152681 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.152659 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b560ce9e-22dd-4f93-8492-ade45b36c42c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.152772 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.152747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vz9\" (UniqueName: \"kubernetes.io/projected/b560ce9e-22dd-4f93-8492-ade45b36c42c-kube-api-access-j8vz9\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cqn78d\" (UID: \"b560ce9e-22dd-4f93-8492-ade45b36c42c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.249518 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.249475 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:02.398294 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:02.398266 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d"] Jul 28 20:06:02.400542 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:06:02.400513 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb560ce9e_22dd_4f93_8492_ade45b36c42c.slice/crio-b7472a2fadd6cec1f1254b9f0a7d82d039589ce3e6c7a625fe7d71c147ac3589 WatchSource:0}: Error finding container b7472a2fadd6cec1f1254b9f0a7d82d039589ce3e6c7a625fe7d71c147ac3589: Status 404 returned error can't find the container with id b7472a2fadd6cec1f1254b9f0a7d82d039589ce3e6c7a625fe7d71c147ac3589 Jul 28 20:06:03.064970 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.064933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" event={"ID":"b560ce9e-22dd-4f93-8492-ade45b36c42c","Type":"ContainerStarted","Data":"b7472a2fadd6cec1f1254b9f0a7d82d039589ce3e6c7a625fe7d71c147ac3589"} Jul 28 20:06:03.698473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.698437 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2"] Jul 28 20:06:03.701899 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.701881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.703923 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.703899 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 20:06:03.704048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.703934 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 20:06:03.704048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.703987 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-njwkk\"" Jul 28 20:06:03.704048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.703988 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 20:06:03.713749 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.713722 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2"] Jul 28 20:06:03.861297 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.861264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.861572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.861303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.861572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.861385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.861572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.861429 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.861572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.861449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.861572 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.861546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbmg\" (UniqueName: \"kubernetes.io/projected/da791d92-4c73-4d7c-85b4-ec422ff63696-kube-api-access-xlbmg\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.861784 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.861620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da791d92-4c73-4d7c-85b4-ec422ff63696-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.962949 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.962840 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da791d92-4c73-4d7c-85b4-ec422ff63696-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.962949 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.962887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.962949 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.962918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.963267 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.962955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.963267 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.963012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.963267 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.963041 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.963267 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.963098 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbmg\" (UniqueName: \"kubernetes.io/projected/da791d92-4c73-4d7c-85b4-ec422ff63696-kube-api-access-xlbmg\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.964830 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.964794 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.965917 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.965894 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/da791d92-4c73-4d7c-85b4-ec422ff63696-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.966247 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.966226 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.966886 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.966866 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.968048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.967988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/da791d92-4c73-4d7c-85b4-ec422ff63696-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.977944 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.977900 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbmg\" (UniqueName: \"kubernetes.io/projected/da791d92-4c73-4d7c-85b4-ec422ff63696-kube-api-access-xlbmg\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:03.978658 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:03.978633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da791d92-4c73-4d7c-85b4-ec422ff63696-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-xckd2\" (UID: \"da791d92-4c73-4d7c-85b4-ec422ff63696\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:04.012694 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:04.012591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:04.207080 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:04.207043 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2"] Jul 28 20:06:04.345796 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:06:04.345763 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda791d92_4c73_4d7c_85b4_ec422ff63696.slice/crio-691f09536c8944b6363ce074c8d6ab319f242b24925ff7f6f0a94b5ae7e19d2a WatchSource:0}: Error finding container 691f09536c8944b6363ce074c8d6ab319f242b24925ff7f6f0a94b5ae7e19d2a: Status 404 returned error can't find the container with id 691f09536c8944b6363ce074c8d6ab319f242b24925ff7f6f0a94b5ae7e19d2a Jul 28 20:06:05.084539 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:05.084494 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" event={"ID":"da791d92-4c73-4d7c-85b4-ec422ff63696","Type":"ContainerStarted","Data":"691f09536c8944b6363ce074c8d6ab319f242b24925ff7f6f0a94b5ae7e19d2a"} Jul 28 20:06:06.503315 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:06.503277 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:06.503609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:06.503366 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:06.503609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:06.503395 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:07.097345 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:07.097258 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" event={"ID":"b560ce9e-22dd-4f93-8492-ade45b36c42c","Type":"ContainerStarted","Data":"bf055b4e560bf7b89320dee20a5533fd9d2796c1182646c91d4e9b13ac2d697c"} Jul 28 20:06:07.119748 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:07.119681 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" podStartSLOduration=2.019728774 podStartE2EDuration="6.119664263s" totalImagesPullingTime="4.099935489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:07.117820094 +0000 UTC m=+394.562131111" watchObservedRunningTime="2026-07-28 20:06:07.119664263 +0000 UTC m=+394.563975282" Jul 28 20:06:07.250106 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:07.250066 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:07.255302 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:07.255274 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:08.102451 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:08.102421 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:08.103598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:08.103577 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cqn78d" Jul 28 20:06:08.283078 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:08.283041 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:08.283143 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:08.283115 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:06:09.108797 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:09.108765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" event={"ID":"da791d92-4c73-4d7c-85b4-ec422ff63696","Type":"ContainerStarted","Data":"f5481a314c7d8233c25d0397e4fda2b79284378ab270110680de3121450f0c60"} Jul 28 20:06:09.109259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:09.108842 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:09.110645 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:09.110625 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" Jul 28 20:06:09.131491 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:06:09.131444 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-xckd2" podStartSLOduration=2.19666637 podStartE2EDuration="6.131422733s" totalImagesPullingTime="3.934756363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:09.130085375 +0000 UTC m=+396.574396390" watchObservedRunningTime="2026-07-28 20:06:09.131422733 +0000 UTC m=+396.575733751" Jul 28 20:07:00.597792 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.597713 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7"] Jul 28 20:07:00.600488 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.600471 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.602577 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.602552 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 20:07:00.602697 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.602641 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dspjd\"" Jul 28 20:07:00.602767 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.602730 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 20:07:00.609037 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.609015 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7"] Jul 28 20:07:00.683710 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.683674 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpwr\" (UniqueName: \"kubernetes.io/projected/38ebac93-002a-4b92-8441-cd3916ea3d07-kube-api-access-xlpwr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.683881 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.683799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.683881 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.683849 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.784842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.784800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.785030 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.784864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.785030 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.784900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpwr\" (UniqueName: \"kubernetes.io/projected/38ebac93-002a-4b92-8441-cd3916ea3d07-kube-api-access-xlpwr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.785193 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.785171 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.785248 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.785207 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.793205 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.793177 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpwr\" (UniqueName: \"kubernetes.io/projected/38ebac93-002a-4b92-8441-cd3916ea3d07-kube-api-access-xlpwr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:00.911043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:00.910964 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:01.041641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.041615 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7"] Jul 28 20:07:01.043583 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:07:01.043557 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ebac93_002a_4b92_8441_cd3916ea3d07.slice/crio-21698c65af05b736ce5ae0473dc29970308a8695b553fc15fed4ba394699c713 WatchSource:0}: Error finding container 21698c65af05b736ce5ae0473dc29970308a8695b553fc15fed4ba394699c713: Status 404 returned error can't find the container with id 21698c65af05b736ce5ae0473dc29970308a8695b553fc15fed4ba394699c713 Jul 28 20:07:01.198682 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.198604 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55"] Jul 28 20:07:01.201449 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.201431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.209691 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.209667 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55"] Jul 28 20:07:01.289304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.289269 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.289496 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.289318 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vg8\" (UniqueName: \"kubernetes.io/projected/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-kube-api-access-t7vg8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.289496 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.289419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.370651 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.370612 2581 generic.go:356] "Generic (PLEG): container finished" podID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerID="19c815b2caf3d7b584e01b13d1558b44658585308e4c2810d25c744bb470c6aa" exitCode=0 Jul 28 20:07:01.370903 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.370660 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" event={"ID":"38ebac93-002a-4b92-8441-cd3916ea3d07","Type":"ContainerDied","Data":"19c815b2caf3d7b584e01b13d1558b44658585308e4c2810d25c744bb470c6aa"} Jul 28 20:07:01.370903 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.370716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" event={"ID":"38ebac93-002a-4b92-8441-cd3916ea3d07","Type":"ContainerStarted","Data":"21698c65af05b736ce5ae0473dc29970308a8695b553fc15fed4ba394699c713"} Jul 28 20:07:01.389997 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.389973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.390107 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.390070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.390153 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.390108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vg8\" (UniqueName: \"kubernetes.io/projected/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-kube-api-access-t7vg8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.390436 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.390410 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.390521 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.390422 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.398206 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.398179 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vg8\" (UniqueName: \"kubernetes.io/projected/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-kube-api-access-t7vg8\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.516868 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.516832 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:01.597379 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.597350 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx"] Jul 28 20:07:01.601136 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.601111 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.608160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.608133 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx"] Jul 28 20:07:01.653035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.653007 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55"] Jul 28 20:07:01.655699 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:07:01.655675 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a5119b_f9d3_4818_91be_1acd26aa5cf0.slice/crio-c75fece93d09ec410855753a08983514a11486fd3556525ff8b4fcfe2077a915 WatchSource:0}: Error finding container c75fece93d09ec410855753a08983514a11486fd3556525ff8b4fcfe2077a915: Status 404 returned error can't find the container with id c75fece93d09ec410855753a08983514a11486fd3556525ff8b4fcfe2077a915 Jul 28 20:07:01.693498 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.693469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.693667 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.693527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8sk\" (UniqueName: \"kubernetes.io/projected/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-kube-api-access-bs8sk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.693667 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.693552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.794587 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.794507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.794729 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.794592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8sk\" (UniqueName: \"kubernetes.io/projected/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-kube-api-access-bs8sk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.794729 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.794615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.794967 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.794949 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.795006 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.794947 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.801905 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.801883 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8sk\" (UniqueName: \"kubernetes.io/projected/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-kube-api-access-bs8sk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:01.913444 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:01.913408 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:02.002235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.002199 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp"] Jul 28 20:07:02.005788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.005765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.014105 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.014063 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp"] Jul 28 20:07:02.048221 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.048197 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx"] Jul 28 20:07:02.050178 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:07:02.050151 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdcb3bc_83fa_4b3e_8a37_17906f715a0a.slice/crio-72bec31ef263755559ff1d9d167d10ca82bafb4415453560152488b4e646cc95 WatchSource:0}: Error finding container 72bec31ef263755559ff1d9d167d10ca82bafb4415453560152488b4e646cc95: Status 404 returned error can't find the container with id 72bec31ef263755559ff1d9d167d10ca82bafb4415453560152488b4e646cc95 Jul 28 20:07:02.097282 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.097256 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.097431 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.097295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqjb\" (UniqueName: \"kubernetes.io/projected/c518a04d-3e69-4cb7-bf5f-3d3566518e95-kube-api-access-rtqjb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.097431 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.097333 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.198384 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.198307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.198549 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.198444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.198549 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.198474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqjb\" (UniqueName: \"kubernetes.io/projected/c518a04d-3e69-4cb7-bf5f-3d3566518e95-kube-api-access-rtqjb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.198766 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.198744 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.198803 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.198753 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.206290 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.206268 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqjb\" (UniqueName: \"kubernetes.io/projected/c518a04d-3e69-4cb7-bf5f-3d3566518e95-kube-api-access-rtqjb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.318732 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.318658 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:02.379481 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.379450 2581 generic.go:356] "Generic (PLEG): container finished" podID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerID="9080bab585654231672e92cdf92cb0b9b16dd273f5b0d9b531629a4b3dbd487b" exitCode=0 Jul 28 20:07:02.379665 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.379533 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" event={"ID":"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a","Type":"ContainerDied","Data":"9080bab585654231672e92cdf92cb0b9b16dd273f5b0d9b531629a4b3dbd487b"} Jul 28 20:07:02.379665 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.379581 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" event={"ID":"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a","Type":"ContainerStarted","Data":"72bec31ef263755559ff1d9d167d10ca82bafb4415453560152488b4e646cc95"} Jul 28 20:07:02.381098 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.381077 2581 generic.go:356] "Generic (PLEG): container finished" podID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerID="9b46514796a456071f18f5eb3841b9ca46d092dc98956a6f2173535b39682f4a" exitCode=0 Jul 28 20:07:02.381199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.381159 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" event={"ID":"a9a5119b-f9d3-4818-91be-1acd26aa5cf0","Type":"ContainerDied","Data":"9b46514796a456071f18f5eb3841b9ca46d092dc98956a6f2173535b39682f4a"} Jul 28 20:07:02.381256 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.381205 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" event={"ID":"a9a5119b-f9d3-4818-91be-1acd26aa5cf0","Type":"ContainerStarted","Data":"c75fece93d09ec410855753a08983514a11486fd3556525ff8b4fcfe2077a915"} Jul 28 20:07:02.545399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:02.545377 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp"] Jul 28 20:07:02.577241 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:07:02.577162 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc518a04d_3e69_4cb7_bf5f_3d3566518e95.slice/crio-f68980a134be2421fafcc9c3650726e9be473222050677442c9a68ea3f1dac40 WatchSource:0}: Error finding container f68980a134be2421fafcc9c3650726e9be473222050677442c9a68ea3f1dac40: Status 404 returned error can't find the container with id f68980a134be2421fafcc9c3650726e9be473222050677442c9a68ea3f1dac40 Jul 28 20:07:03.388408 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:03.388313 2581 generic.go:356] "Generic (PLEG): container finished" podID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerID="23e1111ece9e81d3b30f645f4941f54b5d5c76658d84571c25e08b04e05d662f" exitCode=0 Jul 28 20:07:03.388408 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:03.388380 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" event={"ID":"38ebac93-002a-4b92-8441-cd3916ea3d07","Type":"ContainerDied","Data":"23e1111ece9e81d3b30f645f4941f54b5d5c76658d84571c25e08b04e05d662f"} Jul 28 20:07:03.390030 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:03.389986 2581 generic.go:356] "Generic (PLEG): container finished" podID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerID="f24ef422b39d39373d42e8264246bf4aad92f54fa072e3b7e6390b862d1a4227" exitCode=0 Jul 28 20:07:03.390107 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:03.390071 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" event={"ID":"c518a04d-3e69-4cb7-bf5f-3d3566518e95","Type":"ContainerDied","Data":"f24ef422b39d39373d42e8264246bf4aad92f54fa072e3b7e6390b862d1a4227"} Jul 28 20:07:03.390163 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:03.390115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" event={"ID":"c518a04d-3e69-4cb7-bf5f-3d3566518e95","Type":"ContainerStarted","Data":"f68980a134be2421fafcc9c3650726e9be473222050677442c9a68ea3f1dac40"} Jul 28 20:07:03.392140 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:03.392109 2581 generic.go:356] "Generic (PLEG): container finished" podID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerID="f8fdf7daa38d018267630efc05a2b0e87a1a0ffc892877eab374b913102ff686" exitCode=0 Jul 28 20:07:03.392238 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:03.392155 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" event={"ID":"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a","Type":"ContainerDied","Data":"f8fdf7daa38d018267630efc05a2b0e87a1a0ffc892877eab374b913102ff686"} Jul 28 20:07:04.399605 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.399510 2581 generic.go:356] "Generic (PLEG): container finished" podID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerID="4638e2d903d39dbf5b88c7a413ec21f85698db55c78c2b67900433cc1892cd46" exitCode=0 Jul 28 20:07:04.400008 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.399602 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" event={"ID":"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a","Type":"ContainerDied","Data":"4638e2d903d39dbf5b88c7a413ec21f85698db55c78c2b67900433cc1892cd46"} Jul 28 20:07:04.401229 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.401204 2581 generic.go:356] "Generic (PLEG): container finished" podID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerID="84e4fcbbd1f7ae6f5fac4048deca7ea09d26645cb1dd28674e52d8ca294d33ae" exitCode=0 Jul 28 20:07:04.401369 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.401285 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" event={"ID":"a9a5119b-f9d3-4818-91be-1acd26aa5cf0","Type":"ContainerDied","Data":"84e4fcbbd1f7ae6f5fac4048deca7ea09d26645cb1dd28674e52d8ca294d33ae"} Jul 28 20:07:04.403420 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.403395 2581 generic.go:356] "Generic (PLEG): container finished" podID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerID="541071de35f0a6906a9266f6e2f46de656ad57d65351914a6836dcc6e0618e6d" exitCode=0 Jul 28 20:07:04.403506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.403466 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" event={"ID":"38ebac93-002a-4b92-8441-cd3916ea3d07","Type":"ContainerDied","Data":"541071de35f0a6906a9266f6e2f46de656ad57d65351914a6836dcc6e0618e6d"} Jul 28 20:07:04.405231 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.405190 2581 generic.go:356] "Generic (PLEG): container finished" podID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerID="f415eafcca8bfe3f9b7c66145693bdd199f4ed64babe8f5040871a4959dfc3d5" exitCode=0 Jul 28 20:07:04.405295 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:04.405231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" event={"ID":"c518a04d-3e69-4cb7-bf5f-3d3566518e95","Type":"ContainerDied","Data":"f415eafcca8bfe3f9b7c66145693bdd199f4ed64babe8f5040871a4959dfc3d5"} Jul 28 20:07:05.412587 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.412550 2581 generic.go:356] "Generic (PLEG): container finished" podID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerID="0be47819f7e5f916ccd20bd83a3d134f7c492b86d1829e666e4c00759eb4c462" exitCode=0 Jul 28 20:07:05.412970 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.412635 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" event={"ID":"a9a5119b-f9d3-4818-91be-1acd26aa5cf0","Type":"ContainerDied","Data":"0be47819f7e5f916ccd20bd83a3d134f7c492b86d1829e666e4c00759eb4c462"} Jul 28 20:07:05.414557 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.414537 2581 generic.go:356] "Generic (PLEG): container finished" podID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerID="a9a629db33d7ffe04892c4e889e3e86fd4fc07843c95f9f7896ede92917fb1ae" exitCode=0 Jul 28 20:07:05.414667 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.414573 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" event={"ID":"c518a04d-3e69-4cb7-bf5f-3d3566518e95","Type":"ContainerDied","Data":"a9a629db33d7ffe04892c4e889e3e86fd4fc07843c95f9f7896ede92917fb1ae"} Jul 28 20:07:05.574619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.574584 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:05.577164 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.577135 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:05.630710 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.630682 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-util\") pod \"38ebac93-002a-4b92-8441-cd3916ea3d07\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " Jul 28 20:07:05.630860 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.630736 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs8sk\" (UniqueName: \"kubernetes.io/projected/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-kube-api-access-bs8sk\") pod \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " Jul 28 20:07:05.630860 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.630761 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-util\") pod \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " Jul 28 20:07:05.630860 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.630786 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlpwr\" (UniqueName: \"kubernetes.io/projected/38ebac93-002a-4b92-8441-cd3916ea3d07-kube-api-access-xlpwr\") pod \"38ebac93-002a-4b92-8441-cd3916ea3d07\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " Jul 28 20:07:05.630860 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.630833 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-bundle\") pod \"38ebac93-002a-4b92-8441-cd3916ea3d07\" (UID: \"38ebac93-002a-4b92-8441-cd3916ea3d07\") " Jul 28 20:07:05.631046 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.630876 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-bundle\") pod \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\" (UID: \"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a\") " Jul 28 20:07:05.631415 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.631388 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-bundle" (OuterVolumeSpecName: "bundle") pod "38ebac93-002a-4b92-8441-cd3916ea3d07" (UID: "38ebac93-002a-4b92-8441-cd3916ea3d07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:05.631569 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.631543 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-bundle" (OuterVolumeSpecName: "bundle") pod "cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" (UID: "cbdcb3bc-83fa-4b3e-8a37-17906f715a0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:05.633635 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.633607 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ebac93-002a-4b92-8441-cd3916ea3d07-kube-api-access-xlpwr" (OuterVolumeSpecName: "kube-api-access-xlpwr") pod "38ebac93-002a-4b92-8441-cd3916ea3d07" (UID: "38ebac93-002a-4b92-8441-cd3916ea3d07"). InnerVolumeSpecName "kube-api-access-xlpwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:07:05.633731 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.633644 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-kube-api-access-bs8sk" (OuterVolumeSpecName: "kube-api-access-bs8sk") pod "cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" (UID: "cbdcb3bc-83fa-4b3e-8a37-17906f715a0a"). InnerVolumeSpecName "kube-api-access-bs8sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:07:05.636700 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.636677 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-util" (OuterVolumeSpecName: "util") pod "cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" (UID: "cbdcb3bc-83fa-4b3e-8a37-17906f715a0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:05.637259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.637238 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-util" (OuterVolumeSpecName: "util") pod "38ebac93-002a-4b92-8441-cd3916ea3d07" (UID: "38ebac93-002a-4b92-8441-cd3916ea3d07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:05.731947 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.731848 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8sk\" (UniqueName: \"kubernetes.io/projected/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-kube-api-access-bs8sk\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:05.731947 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.731880 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-util\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:05.731947 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.731890 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlpwr\" (UniqueName: \"kubernetes.io/projected/38ebac93-002a-4b92-8441-cd3916ea3d07-kube-api-access-xlpwr\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:05.731947 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.731899 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:05.731947 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.731908 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbdcb3bc-83fa-4b3e-8a37-17906f715a0a-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:05.731947 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:05.731916 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38ebac93-002a-4b92-8441-cd3916ea3d07-util\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:06.422417 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.422387 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" Jul 28 20:07:06.422870 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.422417 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7" event={"ID":"38ebac93-002a-4b92-8441-cd3916ea3d07","Type":"ContainerDied","Data":"21698c65af05b736ce5ae0473dc29970308a8695b553fc15fed4ba394699c713"} Jul 28 20:07:06.422870 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.422465 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21698c65af05b736ce5ae0473dc29970308a8695b553fc15fed4ba394699c713" Jul 28 20:07:06.424264 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.424244 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" Jul 28 20:07:06.424393 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.424281 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx" event={"ID":"cbdcb3bc-83fa-4b3e-8a37-17906f715a0a","Type":"ContainerDied","Data":"72bec31ef263755559ff1d9d167d10ca82bafb4415453560152488b4e646cc95"} Jul 28 20:07:06.424393 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.424334 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72bec31ef263755559ff1d9d167d10ca82bafb4415453560152488b4e646cc95" Jul 28 20:07:06.559986 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.559964 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:06.591878 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.591851 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:06.638778 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.638745 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7vg8\" (UniqueName: \"kubernetes.io/projected/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-kube-api-access-t7vg8\") pod \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " Jul 28 20:07:06.638954 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.638784 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-bundle\") pod \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " Jul 28 20:07:06.638954 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.638871 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-util\") pod \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\" (UID: \"a9a5119b-f9d3-4818-91be-1acd26aa5cf0\") " Jul 28 20:07:06.638954 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.638922 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtqjb\" (UniqueName: \"kubernetes.io/projected/c518a04d-3e69-4cb7-bf5f-3d3566518e95-kube-api-access-rtqjb\") pod \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " Jul 28 20:07:06.639129 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.638959 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-bundle\") pod \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " Jul 28 20:07:06.639129 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.638993 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-util\") pod \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\" (UID: \"c518a04d-3e69-4cb7-bf5f-3d3566518e95\") " Jul 28 20:07:06.639662 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.639624 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-bundle" (OuterVolumeSpecName: "bundle") pod "c518a04d-3e69-4cb7-bf5f-3d3566518e95" (UID: "c518a04d-3e69-4cb7-bf5f-3d3566518e95"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:06.639886 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.639849 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-bundle" (OuterVolumeSpecName: "bundle") pod "a9a5119b-f9d3-4818-91be-1acd26aa5cf0" (UID: "a9a5119b-f9d3-4818-91be-1acd26aa5cf0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:06.641088 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.641063 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-kube-api-access-t7vg8" (OuterVolumeSpecName: "kube-api-access-t7vg8") pod "a9a5119b-f9d3-4818-91be-1acd26aa5cf0" (UID: "a9a5119b-f9d3-4818-91be-1acd26aa5cf0"). InnerVolumeSpecName "kube-api-access-t7vg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:07:06.641368 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.641345 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c518a04d-3e69-4cb7-bf5f-3d3566518e95-kube-api-access-rtqjb" (OuterVolumeSpecName: "kube-api-access-rtqjb") pod "c518a04d-3e69-4cb7-bf5f-3d3566518e95" (UID: "c518a04d-3e69-4cb7-bf5f-3d3566518e95"). InnerVolumeSpecName "kube-api-access-rtqjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:07:06.645193 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.645171 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-util" (OuterVolumeSpecName: "util") pod "c518a04d-3e69-4cb7-bf5f-3d3566518e95" (UID: "c518a04d-3e69-4cb7-bf5f-3d3566518e95"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:06.645743 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.645721 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-util" (OuterVolumeSpecName: "util") pod "a9a5119b-f9d3-4818-91be-1acd26aa5cf0" (UID: "a9a5119b-f9d3-4818-91be-1acd26aa5cf0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:07:06.740700 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.740664 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqjb\" (UniqueName: \"kubernetes.io/projected/c518a04d-3e69-4cb7-bf5f-3d3566518e95-kube-api-access-rtqjb\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:06.740700 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.740695 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:06.740700 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.740705 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c518a04d-3e69-4cb7-bf5f-3d3566518e95-util\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:06.740912 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.740715 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7vg8\" (UniqueName: \"kubernetes.io/projected/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-kube-api-access-t7vg8\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:06.740912 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.740724 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:06.740912 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:06.740732 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9a5119b-f9d3-4818-91be-1acd26aa5cf0-util\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:07.430723 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:07.430648 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" Jul 28 20:07:07.430723 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:07.430665 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55" event={"ID":"a9a5119b-f9d3-4818-91be-1acd26aa5cf0","Type":"ContainerDied","Data":"c75fece93d09ec410855753a08983514a11486fd3556525ff8b4fcfe2077a915"} Jul 28 20:07:07.430723 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:07.430714 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c75fece93d09ec410855753a08983514a11486fd3556525ff8b4fcfe2077a915" Jul 28 20:07:07.432738 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:07.432711 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" event={"ID":"c518a04d-3e69-4cb7-bf5f-3d3566518e95","Type":"ContainerDied","Data":"f68980a134be2421fafcc9c3650726e9be473222050677442c9a68ea3f1dac40"} Jul 28 20:07:07.432875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:07.432755 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f68980a134be2421fafcc9c3650726e9be473222050677442c9a68ea3f1dac40" Jul 28 20:07:07.432875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:07.432794 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp" Jul 28 20:07:11.143956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.143920 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c4b669769-nxc5w"] Jul 28 20:07:11.144351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144286 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerName="extract" Jul 28 20:07:11.144351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144297 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerName="extract" Jul 28 20:07:11.144351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144306 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerName="pull" Jul 28 20:07:11.144351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144311 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerName="pull" Jul 28 20:07:11.144351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144338 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerName="util" Jul 28 20:07:11.144351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144343 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerName="util" Jul 28 20:07:11.144351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144355 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144361 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144372 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerName="pull" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144380 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerName="pull" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144396 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerName="util" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144403 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerName="util" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144412 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerName="pull" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144417 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerName="pull" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144423 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerName="pull" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144428 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerName="pull" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144434 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerName="util" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144441 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerName="util" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144449 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144454 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144466 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerName="util" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144472 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerName="util" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144480 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144488 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144545 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ebac93-002a-4b92-8441-cd3916ea3d07" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144555 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbdcb3bc-83fa-4b3e-8a37-17906f715a0a" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144565 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9a5119b-f9d3-4818-91be-1acd26aa5cf0" containerName="extract" Jul 28 20:07:11.144686 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.144575 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c518a04d-3e69-4cb7-bf5f-3d3566518e95" containerName="extract" Jul 28 20:07:11.147944 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.147922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.164446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.164419 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4b669769-nxc5w"] Jul 28 20:07:11.282706 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.282662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-trusted-ca-bundle\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.282922 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.282795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-oauth-config\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.282922 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.282841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-service-ca\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.282922 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.282879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-config\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.282922 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.282903 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzlff\" (UniqueName: \"kubernetes.io/projected/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-kube-api-access-kzlff\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.283122 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.282997 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-serving-cert\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.283122 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.283018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-oauth-serving-cert\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.384122 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-oauth-config\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.384300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-service-ca\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.384300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-config\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.384300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzlff\" (UniqueName: \"kubernetes.io/projected/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-kube-api-access-kzlff\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.384300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-serving-cert\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.384300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-oauth-serving-cert\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.384300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384296 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-trusted-ca-bundle\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.385030 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.384998 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-service-ca\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.385133 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.385008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-oauth-serving-cert\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.385133 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.385008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-config\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.385133 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.385128 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-trusted-ca-bundle\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.386552 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.386520 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-serving-cert\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.386647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.386590 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-console-oauth-config\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.391873 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.391855 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzlff\" (UniqueName: \"kubernetes.io/projected/cc25ee02-d4ec-44b6-8262-44ab0cd0cc72-kube-api-access-kzlff\") pod \"console-c4b669769-nxc5w\" (UID: \"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72\") " pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.464899 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.464814 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:11.599023 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:11.598999 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4b669769-nxc5w"] Jul 28 20:07:11.600822 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:07:11.600792 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc25ee02_d4ec_44b6_8262_44ab0cd0cc72.slice/crio-de5906179442526fbddd88aa69501c4990cb5de9b5afe7a820ba04ab0c2c4f9e WatchSource:0}: Error finding container de5906179442526fbddd88aa69501c4990cb5de9b5afe7a820ba04ab0c2c4f9e: Status 404 returned error can't find the container with id de5906179442526fbddd88aa69501c4990cb5de9b5afe7a820ba04ab0c2c4f9e Jul 28 20:07:12.459951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:12.459917 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4b669769-nxc5w" event={"ID":"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72","Type":"ContainerStarted","Data":"d6efd8bb879416ebcdabc265e2f980583e4e73dce163a0851c6ec4dd9f46624c"} Jul 28 20:07:12.460349 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:12.459968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4b669769-nxc5w" event={"ID":"cc25ee02-d4ec-44b6-8262-44ab0cd0cc72","Type":"ContainerStarted","Data":"de5906179442526fbddd88aa69501c4990cb5de9b5afe7a820ba04ab0c2c4f9e"} Jul 28 20:07:12.477560 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:12.477508 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c4b669769-nxc5w" podStartSLOduration=1.477493519 podStartE2EDuration="1.477493519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:12.475397946 +0000 UTC m=+459.919708962" watchObservedRunningTime="2026-07-28 20:07:12.477493519 +0000 UTC m=+459.921804536" Jul 28 20:07:21.465386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:21.465336 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:21.465837 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:21.465428 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:21.470377 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:21.470351 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:21.512259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:21.512230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c4b669769-nxc5w" Jul 28 20:07:21.569492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:21.569461 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777c7df6fd-kt2db"] Jul 28 20:07:46.593304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.593260 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-777c7df6fd-kt2db" podUID="681fdd0a-ba30-4750-83a9-8b7280b02b0a" containerName="console" containerID="cri-o://df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3" gracePeriod=15 Jul 28 20:07:46.833219 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.833192 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:07:46.915014 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.914918 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-oauth-serving-cert\") pod \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " Jul 28 20:07:46.915014 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.914983 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78bq5\" (UniqueName: \"kubernetes.io/projected/681fdd0a-ba30-4750-83a9-8b7280b02b0a-kube-api-access-78bq5\") pod \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " Jul 28 20:07:46.915233 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915059 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-oauth-config\") pod \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " Jul 28 20:07:46.915233 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915106 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-trusted-ca-bundle\") pod \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " Jul 28 20:07:46.915233 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915131 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-serving-cert\") pod \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " Jul 28 20:07:46.915233 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915176 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-service-ca\") pod \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " Jul 28 20:07:46.915233 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915200 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-config\") pod \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\" (UID: \"681fdd0a-ba30-4750-83a9-8b7280b02b0a\") " Jul 28 20:07:46.915502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915297 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "681fdd0a-ba30-4750-83a9-8b7280b02b0a" (UID: "681fdd0a-ba30-4750-83a9-8b7280b02b0a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:07:46.915556 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915509 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-oauth-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:46.915683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915570 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-service-ca" (OuterVolumeSpecName: "service-ca") pod "681fdd0a-ba30-4750-83a9-8b7280b02b0a" (UID: "681fdd0a-ba30-4750-83a9-8b7280b02b0a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:07:46.915683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915662 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "681fdd0a-ba30-4750-83a9-8b7280b02b0a" (UID: "681fdd0a-ba30-4750-83a9-8b7280b02b0a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:07:46.915971 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.915726 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-config" (OuterVolumeSpecName: "console-config") pod "681fdd0a-ba30-4750-83a9-8b7280b02b0a" (UID: "681fdd0a-ba30-4750-83a9-8b7280b02b0a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:07:46.917189 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.917168 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "681fdd0a-ba30-4750-83a9-8b7280b02b0a" (UID: "681fdd0a-ba30-4750-83a9-8b7280b02b0a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:07:46.917631 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.917612 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681fdd0a-ba30-4750-83a9-8b7280b02b0a-kube-api-access-78bq5" (OuterVolumeSpecName: "kube-api-access-78bq5") pod "681fdd0a-ba30-4750-83a9-8b7280b02b0a" (UID: "681fdd0a-ba30-4750-83a9-8b7280b02b0a"). InnerVolumeSpecName "kube-api-access-78bq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:07:46.917692 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:46.917612 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "681fdd0a-ba30-4750-83a9-8b7280b02b0a" (UID: "681fdd0a-ba30-4750-83a9-8b7280b02b0a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:07:47.016825 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.016778 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bq5\" (UniqueName: \"kubernetes.io/projected/681fdd0a-ba30-4750-83a9-8b7280b02b0a-kube-api-access-78bq5\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:47.016825 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.016815 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-oauth-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:47.016825 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.016829 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-trusted-ca-bundle\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:47.017066 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.016840 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-serving-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:47.017066 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.016853 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:47.017066 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.016864 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/681fdd0a-ba30-4750-83a9-8b7280b02b0a-console-config\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:07:47.644220 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.644179 2581 generic.go:356] "Generic (PLEG): container finished" podID="681fdd0a-ba30-4750-83a9-8b7280b02b0a" containerID="df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3" exitCode=0 Jul 28 20:07:47.644220 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.644226 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-777c7df6fd-kt2db" Jul 28 20:07:47.644736 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.644223 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777c7df6fd-kt2db" event={"ID":"681fdd0a-ba30-4750-83a9-8b7280b02b0a","Type":"ContainerDied","Data":"df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3"} Jul 28 20:07:47.644736 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.644368 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-777c7df6fd-kt2db" event={"ID":"681fdd0a-ba30-4750-83a9-8b7280b02b0a","Type":"ContainerDied","Data":"0cfbeb8267abed2985895d89a3252f05e22be79debc45c12185abb3e84792c7f"} Jul 28 20:07:47.644736 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.644395 2581 scope.go:117] "RemoveContainer" containerID="df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3" Jul 28 20:07:47.653688 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.653672 2581 scope.go:117] "RemoveContainer" containerID="df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3" Jul 28 20:07:47.653932 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:07:47.653913 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3\": container with ID starting with df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3 not found: ID does not exist" containerID="df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3" Jul 28 20:07:47.653996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.653944 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3"} err="failed to get container status \"df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3\": rpc error: code = NotFound desc = could not find container \"df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3\": container with ID starting with df1f0630db7ccbe42da19f307ece95d844a4fc39b69665cba2a71e2d4f1923e3 not found: ID does not exist" Jul 28 20:07:47.661156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.661135 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-777c7df6fd-kt2db"] Jul 28 20:07:47.665556 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:47.665537 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-777c7df6fd-kt2db"] Jul 28 20:07:49.157597 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:07:49.152618 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="681fdd0a-ba30-4750-83a9-8b7280b02b0a" path="/var/lib/kubelet/pods/681fdd0a-ba30-4750-83a9-8b7280b02b0a/volumes" Jul 28 20:08:48.325924 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.325892 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:08:48.326461 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.326355 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="681fdd0a-ba30-4750-83a9-8b7280b02b0a" containerName="console" Jul 28 20:08:48.326461 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.326370 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="681fdd0a-ba30-4750-83a9-8b7280b02b0a" containerName="console" Jul 28 20:08:48.326461 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.326442 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="681fdd0a-ba30-4750-83a9-8b7280b02b0a" containerName="console" Jul 28 20:08:48.329060 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.329045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.331241 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.331220 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 20:08:48.331393 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.331349 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 20:08:48.331819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.331806 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dspjd\"" Jul 28 20:08:48.331819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.331815 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 20:08:48.336957 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.336936 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:08:48.369064 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.369033 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:08:48.459711 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.459672 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8x7\" (UniqueName: \"kubernetes.io/projected/7c93be44-5de6-43f6-9ff0-2f9f7124fb7d-kube-api-access-cg8x7\") pod \"limitador-limitador-54d6d988d5-tth4c\" (UID: \"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.459891 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.459806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c93be44-5de6-43f6-9ff0-2f9f7124fb7d-config-file\") pod \"limitador-limitador-54d6d988d5-tth4c\" (UID: \"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.561132 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.561070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c93be44-5de6-43f6-9ff0-2f9f7124fb7d-config-file\") pod \"limitador-limitador-54d6d988d5-tth4c\" (UID: \"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.561132 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.561147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8x7\" (UniqueName: \"kubernetes.io/projected/7c93be44-5de6-43f6-9ff0-2f9f7124fb7d-kube-api-access-cg8x7\") pod \"limitador-limitador-54d6d988d5-tth4c\" (UID: \"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.561786 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.561764 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c93be44-5de6-43f6-9ff0-2f9f7124fb7d-config-file\") pod \"limitador-limitador-54d6d988d5-tth4c\" (UID: \"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.569061 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.569038 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8x7\" (UniqueName: \"kubernetes.io/projected/7c93be44-5de6-43f6-9ff0-2f9f7124fb7d-kube-api-access-cg8x7\") pod \"limitador-limitador-54d6d988d5-tth4c\" (UID: \"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.640487 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.640401 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:48.767343 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.767302 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:08:48.769051 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:08:48.769021 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c93be44_5de6_43f6_9ff0_2f9f7124fb7d.slice/crio-dab6f3a8a47ca2942666f1bb31f8ed7902b0533a25039013682188b4196f3899 WatchSource:0}: Error finding container dab6f3a8a47ca2942666f1bb31f8ed7902b0533a25039013682188b4196f3899: Status 404 returned error can't find the container with id dab6f3a8a47ca2942666f1bb31f8ed7902b0533a25039013682188b4196f3899 Jul 28 20:08:48.962075 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:48.961994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" event={"ID":"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d","Type":"ContainerStarted","Data":"dab6f3a8a47ca2942666f1bb31f8ed7902b0533a25039013682188b4196f3899"} Jul 28 20:08:51.981973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:51.981931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" event={"ID":"7c93be44-5de6-43f6-9ff0-2f9f7124fb7d","Type":"ContainerStarted","Data":"b482add637b1ca27d5f79f9de56a76f6e2bc80bd2b4420e83cf2f098c943b89f"} Jul 28 20:08:51.982436 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:51.981999 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:08:51.997442 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:08:51.997379 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" podStartSLOduration=1.329192412 podStartE2EDuration="3.997365254s" totalImagesPullingTime="2.668172842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:08:51.995358157 +0000 UTC m=+559.439669172" watchObservedRunningTime="2026-07-28 20:08:51.997365254 +0000 UTC m=+559.441676270" Jul 28 20:09:02.989124 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:09:02.989096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tth4c" Jul 28 20:10:06.105585 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:10:06.105497 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:10:44.694581 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:10:44.694545 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:10:58.381917 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:10:58.381876 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:11:06.385692 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:11:06.385652 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:11:16.304360 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:11:16.304317 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:11:43.485291 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:11:43.485254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:11:56.581583 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:11:56.581545 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:13:25.075084 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:13:25.075048 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:13:34.579091 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:13:34.579055 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:13:45.777688 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:13:45.777652 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:15:00.145076 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.145038 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf"] Jul 28 20:15:00.148382 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.148358 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf" Jul 28 20:15:00.150756 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.150735 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:15:00.154903 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.154877 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf"] Jul 28 20:15:00.160925 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.160907 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf" Jul 28 20:15:00.498197 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.498170 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf"] Jul 28 20:15:00.499597 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:15:00.499571 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552a8af6_9749_4128_bd82_83cacf51515a.slice/crio-4f52c11e4c23464a09a07d5bc2f6cd5be4725aae904b78193452bf86fe338fee WatchSource:0}: Error finding container 4f52c11e4c23464a09a07d5bc2f6cd5be4725aae904b78193452bf86fe338fee: Status 404 returned error can't find the container with id 4f52c11e4c23464a09a07d5bc2f6cd5be4725aae904b78193452bf86fe338fee Jul 28 20:15:00.501487 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.501467 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:15:00.892174 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:00.892084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf" event={"ID":"552a8af6-9749-4128-bd82-83cacf51515a","Type":"ContainerStarted","Data":"4f52c11e4c23464a09a07d5bc2f6cd5be4725aae904b78193452bf86fe338fee"} Jul 28 20:15:03.912035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:03.912000 2581 generic.go:356] "Generic (PLEG): container finished" podID="552a8af6-9749-4128-bd82-83cacf51515a" containerID="5c84883be114fe4d7e27504518b2c9191a29369a3ac0450f67a2a0e76af47e97" exitCode=0 Jul 28 20:15:03.912383 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:03.912082 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf" event={"ID":"552a8af6-9749-4128-bd82-83cacf51515a","Type":"ContainerDied","Data":"5c84883be114fe4d7e27504518b2c9191a29369a3ac0450f67a2a0e76af47e97"} Jul 28 20:15:05.049816 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:05.049792 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf" Jul 28 20:15:05.925047 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:05.925007 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf" event={"ID":"552a8af6-9749-4128-bd82-83cacf51515a","Type":"ContainerDied","Data":"4f52c11e4c23464a09a07d5bc2f6cd5be4725aae904b78193452bf86fe338fee"} Jul 28 20:15:05.925047 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:05.925036 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf" Jul 28 20:15:05.925336 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:05.925063 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f52c11e4c23464a09a07d5bc2f6cd5be4725aae904b78193452bf86fe338fee" Jul 28 20:15:24.897104 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:24.897069 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:15:35.585177 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:15:35.585144 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:16:02.987600 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:16:02.987509 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:16:12.879944 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:16:12.879913 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:16:22.278492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:16:22.278451 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:16:33.489235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:16:33.489196 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:16:45.076291 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:16:45.076256 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:16:55.922211 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:16:55.922175 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:17:08.274255 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:17:08.274220 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:17:18.075230 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:17:18.075194 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:17:39.779096 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:17:39.779059 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:17:49.072103 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:17:49.072065 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:18:11.876440 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:18:11.876405 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:18:22.580442 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:18:22.580406 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:18:32.573938 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:18:32.573898 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:18:48.982858 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:18:48.982820 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:18:58.488820 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:18:58.488728 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:19:08.681493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:19:08.681456 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:19:20.696961 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:19:20.696923 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:19:57.792770 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:19:57.792730 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:21:25.499254 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:21:25.499215 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:21:37.921198 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:21:37.921163 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:21:54.950012 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:21:54.949889 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:22:05.210912 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:22:05.210876 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:22:17.922821 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:22:17.922778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:22:25.725730 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:22:25.725693 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:22:37.250088 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:22:37.250048 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:22:48.101030 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:22:48.100990 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:22:55.840455 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:22:55.840418 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:23:07.893622 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:23:07.893566 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:23:20.285358 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:23:20.285309 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:23:31.386086 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:23:31.386012 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:23:46.988216 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:23:46.988181 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:23:54.691294 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:23:54.691254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:24:07.605956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:24:07.605918 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:24:19.086619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:24:19.086578 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:24:39.388626 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:24:39.388591 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:24:53.494961 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:24:53.494924 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:24:59.995610 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:24:59.995530 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:25:16.416928 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:25:16.416896 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:25:30.875065 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:25:30.875034 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:25:40.088614 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:25:40.088575 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:25:54.192880 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:25:54.192847 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:26:03.583175 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:26:03.583138 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:26:16.811375 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:26:16.811339 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:26:26.308154 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:26:26.308119 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:26:36.003676 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:26:36.003631 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:26:49.283370 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:26:49.283332 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:26:58.887386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:26:58.887353 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:27:09.093393 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:27:09.093353 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:27:22.087421 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:27:22.087383 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:27:36.398691 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:27:36.398653 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:27:45.699075 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:27:45.699038 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:27:56.822574 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:27:56.822487 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:28:10.416152 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:10.416117 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:28:20.791650 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:20.791610 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:28:30.996688 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:30.996651 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n"] Jul 28 20:28:30.997076 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:30.997059 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552a8af6-9749-4128-bd82-83cacf51515a" containerName="cleanup" Jul 28 20:28:30.997076 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:30.997071 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="552a8af6-9749-4128-bd82-83cacf51515a" containerName="cleanup" Jul 28 20:28:30.997166 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:30.997160 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="552a8af6-9749-4128-bd82-83cacf51515a" containerName="cleanup" Jul 28 20:28:31.000442 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.000421 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.005592 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.005570 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a4f1a2e9-openshift-default-dockercfg-wq94t\"" Jul 28 20:28:31.019536 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-credential-socket\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019630 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-certs\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019630 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019556 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-token\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019630 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-data\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019738 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7051ef0a-9b01-451c-866f-4ea4a978ad25-istiod-ca-cert\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019738 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019682 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-envoy\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019738 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-podinfo\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019831 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019753 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6rcr\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-kube-api-access-f6rcr\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.019831 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.019786 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-socket\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.030400 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.030369 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n"] Jul 28 20:28:31.120522 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7051ef0a-9b01-451c-866f-4ea4a978ad25-istiod-ca-cert\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.120522 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-envoy\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.120759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120573 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-podinfo\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.120759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6rcr\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-kube-api-access-f6rcr\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.120759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-socket\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.120759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-credential-socket\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.120759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-certs\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.120759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-token\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.121069 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.120766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-data\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.121134 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.121112 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-credential-socket\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.121189 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.121163 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-data\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.121266 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.121241 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-certs\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.121349 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.121305 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7051ef0a-9b01-451c-866f-4ea4a978ad25-istiod-ca-cert\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.121494 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.121475 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-socket\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.122885 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.122863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-envoy\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.123216 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.123196 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-podinfo\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.131956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.131934 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-token\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.132229 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.132207 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6rcr\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-kube-api-access-f6rcr\") pod \"e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.263860 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.263761 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:28:31.310557 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.310525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:31.489168 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.489141 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n"] Jul 28 20:28:31.491014 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:28:31.490987 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7051ef0a_9b01_451c_866f_4ea4a978ad25.slice/crio-64cf90658335d7e2b9430bd28f73134a257a2d1f98661f21fe3e21c57605ff8a WatchSource:0}: Error finding container 64cf90658335d7e2b9430bd28f73134a257a2d1f98661f21fe3e21c57605ff8a: Status 404 returned error can't find the container with id 64cf90658335d7e2b9430bd28f73134a257a2d1f98661f21fe3e21c57605ff8a Jul 28 20:28:31.493079 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.493047 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:28:31.493142 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.493109 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:28:31.493142 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:31.493137 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:28:32.247771 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:32.247735 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" event={"ID":"7051ef0a-9b01-451c-866f-4ea4a978ad25","Type":"ContainerStarted","Data":"f06a59c3de9cb4bdca832acd2c341941206eaaa7f18c18db66f740a0d2445163"} Jul 28 20:28:32.248171 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:32.247797 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" event={"ID":"7051ef0a-9b01-451c-866f-4ea4a978ad25","Type":"ContainerStarted","Data":"64cf90658335d7e2b9430bd28f73134a257a2d1f98661f21fe3e21c57605ff8a"} Jul 28 20:28:32.272831 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:32.272763 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" podStartSLOduration=2.272744668 podStartE2EDuration="2.272744668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:32.268292167 +0000 UTC m=+1739.712603184" watchObservedRunningTime="2026-07-28 20:28:32.272744668 +0000 UTC m=+1739.717055684" Jul 28 20:28:32.311447 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:32.311316 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:32.316541 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:32.316511 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:33.253384 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:33.253356 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:33.254426 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:33.254407 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:39.985801 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:39.985765 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:28:48.255217 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.255176 2581 patch_prober.go:28] interesting pod/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 28 20:28:48.255668 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.255234 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 28 20:28:48.340820 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.340789 2581 generic.go:356] "Generic (PLEG): container finished" podID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerID="f06a59c3de9cb4bdca832acd2c341941206eaaa7f18c18db66f740a0d2445163" exitCode=0 Jul 28 20:28:48.341007 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.340863 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" event={"ID":"7051ef0a-9b01-451c-866f-4ea4a978ad25","Type":"ContainerDied","Data":"f06a59c3de9cb4bdca832acd2c341941206eaaa7f18c18db66f740a0d2445163"} Jul 28 20:28:48.341275 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.341261 2581 scope.go:117] "RemoveContainer" containerID="f06a59c3de9cb4bdca832acd2c341941206eaaa7f18c18db66f740a0d2445163" Jul 28 20:28:48.342409 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.342373 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:28:48.342559 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.342458 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:28:48.342559 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:48.342530 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:28:49.347747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:49.347712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" event={"ID":"7051ef0a-9b01-451c-866f-4ea4a978ad25","Type":"ContainerStarted","Data":"1bddaf6bb6484c6715e20f35a0160c0cb860befaae30ec7539a87c636b25f0f9"} Jul 28 20:28:50.311801 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:50.311765 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:50.312173 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:50.312093 2581 patch_prober.go:28] interesting pod/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 28 20:28:50.312340 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:50.312297 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 28 20:28:51.311683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:51.311646 2581 patch_prober.go:28] interesting pod/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 28 20:28:51.312314 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:51.312280 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 28 20:28:52.311436 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:52.311399 2581 patch_prober.go:28] interesting pod/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 28 20:28:52.311747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:52.311718 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 28 20:28:53.315484 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:53.315452 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:53.315853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:53.315713 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:53.316422 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:53.316403 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:28:59.185609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.185577 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t"] Jul 28 20:28:59.188009 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.187991 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" Jul 28 20:28:59.190576 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.190548 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:28:59.190675 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.190591 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 20:28:59.197743 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.197720 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t"] Jul 28 20:28:59.299169 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.299136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f915c8-4ba4-4ee2-bfe5-9b43c8699bef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t\" (UID: \"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" Jul 28 20:28:59.400340 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.400300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f915c8-4ba4-4ee2-bfe5-9b43c8699bef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t\" (UID: \"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" Jul 28 20:28:59.400891 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.400870 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f915c8-4ba4-4ee2-bfe5-9b43c8699bef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t\" (UID: \"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" Jul 28 20:28:59.499139 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.499106 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" Jul 28 20:28:59.630317 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:28:59.630289 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t"] Jul 28 20:28:59.633063 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:28:59.632915 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f915c8_4ba4_4ee2_bfe5_9b43c8699bef.slice/crio-2e609bb19b64ab6287b3152ffb8bb2a4db7577094a11a46dd69ac7651ff3d05e WatchSource:0}: Error finding container 2e609bb19b64ab6287b3152ffb8bb2a4db7577094a11a46dd69ac7651ff3d05e: Status 404 returned error can't find the container with id 2e609bb19b64ab6287b3152ffb8bb2a4db7577094a11a46dd69ac7651ff3d05e Jul 28 20:29:00.408728 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:00.408692 2581 generic.go:356] "Generic (PLEG): container finished" podID="84f915c8-4ba4-4ee2-bfe5-9b43c8699bef" containerID="29994152a017f98af03ef0b32520b7c886488b887809ddf14570e85c2c88b176" exitCode=0 Jul 28 20:29:00.409091 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:00.408764 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" event={"ID":"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef","Type":"ContainerDied","Data":"29994152a017f98af03ef0b32520b7c886488b887809ddf14570e85c2c88b176"} Jul 28 20:29:00.409091 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:00.408845 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" event={"ID":"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef","Type":"ContainerStarted","Data":"2e609bb19b64ab6287b3152ffb8bb2a4db7577094a11a46dd69ac7651ff3d05e"} Jul 28 20:29:01.546294 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:01.546271 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" Jul 28 20:29:01.721630 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:01.721519 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f915c8-4ba4-4ee2-bfe5-9b43c8699bef-maas-api-service-ca\") pod \"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef\" (UID: \"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef\") " Jul 28 20:29:01.721939 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:01.721913 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f915c8-4ba4-4ee2-bfe5-9b43c8699bef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84f915c8-4ba4-4ee2-bfe5-9b43c8699bef" (UID: "84f915c8-4ba4-4ee2-bfe5-9b43c8699bef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:29:01.822250 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:01.822210 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f915c8-4ba4-4ee2-bfe5-9b43c8699bef-maas-api-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:02.421391 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:02.421358 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" Jul 28 20:29:02.421578 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:02.421392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t" event={"ID":"84f915c8-4ba4-4ee2-bfe5-9b43c8699bef","Type":"ContainerDied","Data":"2e609bb19b64ab6287b3152ffb8bb2a4db7577094a11a46dd69ac7651ff3d05e"} Jul 28 20:29:02.421578 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:02.421447 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e609bb19b64ab6287b3152ffb8bb2a4db7577094a11a46dd69ac7651ff3d05e" Jul 28 20:29:05.096112 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:05.096078 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t"] Jul 28 20:29:05.105846 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:05.105816 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4f1a2e9-xch4t"] Jul 28 20:29:05.152132 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:05.152099 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f915c8-4ba4-4ee2-bfe5-9b43c8699bef" path="/var/lib/kubelet/pods/84f915c8-4ba4-4ee2-bfe5-9b43c8699bef/volumes" Jul 28 20:29:05.282540 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:05.282503 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n"] Jul 28 20:29:05.282772 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:05.282748 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" containerID="cri-o://1bddaf6bb6484c6715e20f35a0160c0cb860befaae30ec7539a87c636b25f0f9" gracePeriod=30 Jul 28 20:29:08.317894 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:08.317853 2581 patch_prober.go:28] interesting pod/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 20:29:08.318276 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:08.317912 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 20:29:08.809658 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:08.809625 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:29:10.468543 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.468504 2581 generic.go:356] "Generic (PLEG): container finished" podID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerID="1bddaf6bb6484c6715e20f35a0160c0cb860befaae30ec7539a87c636b25f0f9" exitCode=0 Jul 28 20:29:10.468941 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.468568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" event={"ID":"7051ef0a-9b01-451c-866f-4ea4a978ad25","Type":"ContainerDied","Data":"1bddaf6bb6484c6715e20f35a0160c0cb860befaae30ec7539a87c636b25f0f9"} Jul 28 20:29:10.468941 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.468687 2581 scope.go:117] "RemoveContainer" containerID="f06a59c3de9cb4bdca832acd2c341941206eaaa7f18c18db66f740a0d2445163" Jul 28 20:29:10.537734 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.537713 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:29:10.603136 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.603060 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-socket\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.603291 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.603262 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:29:10.704123 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704088 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-token\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704295 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704131 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-podinfo\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704295 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704149 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-certs\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704295 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704179 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6rcr\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-kube-api-access-f6rcr\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704295 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704210 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7051ef0a-9b01-451c-866f-4ea4a978ad25-istiod-ca-cert\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704295 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704255 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-envoy\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704600 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704297 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-credential-socket\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704600 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704370 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-data\") pod \"7051ef0a-9b01-451c-866f-4ea4a978ad25\" (UID: \"7051ef0a-9b01-451c-866f-4ea4a978ad25\") " Jul 28 20:29:10.704600 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704497 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:29:10.704768 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704658 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.704768 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704677 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-workload-certs\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.704768 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704692 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7051ef0a-9b01-451c-866f-4ea4a978ad25-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:29:10.704768 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.704718 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:29:10.705269 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.705244 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-data" (OuterVolumeSpecName: "istio-data") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:29:10.706681 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.706651 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:29:10.706798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.706717 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-kube-api-access-f6rcr" (OuterVolumeSpecName: "kube-api-access-f6rcr") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "kube-api-access-f6rcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:29:10.706798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.706774 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-token" (OuterVolumeSpecName: "istio-token") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:29:10.706957 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.706941 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7051ef0a-9b01-451c-866f-4ea4a978ad25" (UID: "7051ef0a-9b01-451c-866f-4ea4a978ad25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:29:10.805856 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.805820 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-token\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.805856 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.805850 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-podinfo\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.805856 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.805860 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6rcr\" (UniqueName: \"kubernetes.io/projected/7051ef0a-9b01-451c-866f-4ea4a978ad25-kube-api-access-f6rcr\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.806097 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.805869 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7051ef0a-9b01-451c-866f-4ea4a978ad25-istiod-ca-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.806097 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.805877 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-envoy\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.806097 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.805885 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-credential-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:10.806097 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:10.805892 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7051ef0a-9b01-451c-866f-4ea4a978ad25-istio-data\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:11.474942 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:11.474859 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" Jul 28 20:29:11.474942 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:11.474907 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n" event={"ID":"7051ef0a-9b01-451c-866f-4ea4a978ad25","Type":"ContainerDied","Data":"64cf90658335d7e2b9430bd28f73134a257a2d1f98661f21fe3e21c57605ff8a"} Jul 28 20:29:11.475446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:11.474963 2581 scope.go:117] "RemoveContainer" containerID="1bddaf6bb6484c6715e20f35a0160c0cb860befaae30ec7539a87c636b25f0f9" Jul 28 20:29:11.491240 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:11.491214 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n"] Jul 28 20:29:11.498009 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:11.497981 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a4f1a2e9-openshift-default-5f675cd4c4-5629n"] Jul 28 20:29:13.152401 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:13.152367 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" path="/var/lib/kubelet/pods/7051ef0a-9b01-451c-866f-4ea4a978ad25/volumes" Jul 28 20:29:21.337057 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:21.337026 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:29:34.367137 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367099 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4"] Jul 28 20:29:34.367545 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367487 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" Jul 28 20:29:34.367545 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367499 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" Jul 28 20:29:34.367545 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367519 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" Jul 28 20:29:34.367545 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367525 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" Jul 28 20:29:34.367545 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367534 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f915c8-4ba4-4ee2-bfe5-9b43c8699bef" containerName="revoke-keys" Jul 28 20:29:34.367545 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367540 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f915c8-4ba4-4ee2-bfe5-9b43c8699bef" containerName="revoke-keys" Jul 28 20:29:34.367758 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367601 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" Jul 28 20:29:34.367758 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.367613 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f915c8-4ba4-4ee2-bfe5-9b43c8699bef" containerName="revoke-keys" Jul 28 20:29:34.369705 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.369689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" Jul 28 20:29:34.372028 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.372009 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 20:29:34.372135 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.372077 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:29:34.378845 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.378822 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4"] Jul 28 20:29:34.522419 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.522385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e013e-eb9e-4d61-9adf-107c1781d6d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4\" (UID: \"423e013e-eb9e-4d61-9adf-107c1781d6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" Jul 28 20:29:34.623545 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.623463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e013e-eb9e-4d61-9adf-107c1781d6d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4\" (UID: \"423e013e-eb9e-4d61-9adf-107c1781d6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" Jul 28 20:29:34.624076 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.624057 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e013e-eb9e-4d61-9adf-107c1781d6d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4\" (UID: \"423e013e-eb9e-4d61-9adf-107c1781d6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" Jul 28 20:29:34.681198 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.681163 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" Jul 28 20:29:34.810160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:34.810133 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4"] Jul 28 20:29:34.811844 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:29:34.811814 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423e013e_eb9e_4d61_9adf_107c1781d6d3.slice/crio-0f3ed933c2360621c1f01cf1f151437b9984b327a3cfd6521273fc0cb77a4de5 WatchSource:0}: Error finding container 0f3ed933c2360621c1f01cf1f151437b9984b327a3cfd6521273fc0cb77a4de5: Status 404 returned error can't find the container with id 0f3ed933c2360621c1f01cf1f151437b9984b327a3cfd6521273fc0cb77a4de5 Jul 28 20:29:35.605126 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:35.605087 2581 generic.go:356] "Generic (PLEG): container finished" podID="423e013e-eb9e-4d61-9adf-107c1781d6d3" containerID="a201595e37157ae4c866685c75f8e7ea11b3d4155088d8731b8b3845d0ece7ba" exitCode=0 Jul 28 20:29:35.605518 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:35.605177 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" event={"ID":"423e013e-eb9e-4d61-9adf-107c1781d6d3","Type":"ContainerDied","Data":"a201595e37157ae4c866685c75f8e7ea11b3d4155088d8731b8b3845d0ece7ba"} Jul 28 20:29:35.605518 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:35.605231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" event={"ID":"423e013e-eb9e-4d61-9adf-107c1781d6d3","Type":"ContainerStarted","Data":"0f3ed933c2360621c1f01cf1f151437b9984b327a3cfd6521273fc0cb77a4de5"} Jul 28 20:29:36.739750 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:36.739726 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" Jul 28 20:29:36.843934 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:36.843884 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e013e-eb9e-4d61-9adf-107c1781d6d3-maas-api-service-ca\") pod \"423e013e-eb9e-4d61-9adf-107c1781d6d3\" (UID: \"423e013e-eb9e-4d61-9adf-107c1781d6d3\") " Jul 28 20:29:36.844289 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:36.844266 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423e013e-eb9e-4d61-9adf-107c1781d6d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "423e013e-eb9e-4d61-9adf-107c1781d6d3" (UID: "423e013e-eb9e-4d61-9adf-107c1781d6d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:29:36.944743 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:36.944651 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e013e-eb9e-4d61-9adf-107c1781d6d3-maas-api-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:29:37.616666 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:37.616629 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" Jul 28 20:29:37.616853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:37.616682 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4" event={"ID":"423e013e-eb9e-4d61-9adf-107c1781d6d3","Type":"ContainerDied","Data":"0f3ed933c2360621c1f01cf1f151437b9984b327a3cfd6521273fc0cb77a4de5"} Jul 28 20:29:37.616853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:37.616779 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f3ed933c2360621c1f01cf1f151437b9984b327a3cfd6521273fc0cb77a4de5" Jul 28 20:29:40.686099 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:40.686070 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4"] Jul 28 20:29:40.692979 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:40.692955 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-addc0a70-z7rx4"] Jul 28 20:29:41.152396 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:41.152362 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423e013e-eb9e-4d61-9adf-107c1781d6d3" path="/var/lib/kubelet/pods/423e013e-eb9e-4d61-9adf-107c1781d6d3/volumes" Jul 28 20:29:42.013595 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.013560 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh"] Jul 28 20:29:42.013972 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.013958 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423e013e-eb9e-4d61-9adf-107c1781d6d3" containerName="revoke-keys" Jul 28 20:29:42.014013 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.013974 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="423e013e-eb9e-4d61-9adf-107c1781d6d3" containerName="revoke-keys" Jul 28 20:29:42.014049 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.014025 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="423e013e-eb9e-4d61-9adf-107c1781d6d3" containerName="revoke-keys" Jul 28 20:29:42.014049 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.014036 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7051ef0a-9b01-451c-866f-4ea4a978ad25" containerName="istio-proxy" Jul 28 20:29:42.016298 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.016282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.018966 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.018941 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-89b18cb8-openshift-default-dockercfg-8z6ks\"" Jul 28 20:29:42.056627 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.056598 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh"] Jul 28 20:29:42.095774 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.095740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-socket\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096104 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096068 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-credential-socket\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096236 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b1170-9496-499c-ad80-aab06e4382a5-istio-podinfo\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096236 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g99v\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-kube-api-access-4g99v\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096236 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-istio-token\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096236 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-certs\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b1170-9496-499c-ad80-aab06e4382a5-istiod-ca-cert\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-data\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.096493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.096371 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-envoy\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197068 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-credential-socket\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197078 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b1170-9496-499c-ad80-aab06e4382a5-istio-podinfo\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g99v\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-kube-api-access-4g99v\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-istio-token\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-certs\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197165 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b1170-9496-499c-ad80-aab06e4382a5-istiod-ca-cert\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197192 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-data\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-envoy\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197644 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-socket\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197768 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197740 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-certs\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197824 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-data\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.197946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.197925 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-credential-socket\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.198215 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.198187 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-socket\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.198292 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.198228 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b1170-9496-499c-ad80-aab06e4382a5-istiod-ca-cert\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.199732 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.199700 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b1170-9496-499c-ad80-aab06e4382a5-istio-podinfo\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.199923 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.199899 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-envoy\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.213754 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.213733 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-istio-token\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.214165 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.214147 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g99v\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-kube-api-access-4g99v\") pod \"e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.326888 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.326792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:42.473505 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.473476 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh"] Jul 28 20:29:42.474308 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:29:42.474282 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861b1170_9496_499c_ad80_aab06e4382a5.slice/crio-5e30be5baf48efb62d112f3f3d49a0aefdb87879b84f3e568be6fe9cff5a1c12 WatchSource:0}: Error finding container 5e30be5baf48efb62d112f3f3d49a0aefdb87879b84f3e568be6fe9cff5a1c12: Status 404 returned error can't find the container with id 5e30be5baf48efb62d112f3f3d49a0aefdb87879b84f3e568be6fe9cff5a1c12 Jul 28 20:29:42.476494 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.476465 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:29:42.476562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.476528 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:29:42.476562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.476555 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:29:42.647264 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.647168 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" event={"ID":"861b1170-9496-499c-ad80-aab06e4382a5","Type":"ContainerStarted","Data":"2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142"} Jul 28 20:29:42.647264 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.647242 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" event={"ID":"861b1170-9496-499c-ad80-aab06e4382a5","Type":"ContainerStarted","Data":"5e30be5baf48efb62d112f3f3d49a0aefdb87879b84f3e568be6fe9cff5a1c12"} Jul 28 20:29:42.676927 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:42.676872 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" podStartSLOduration=1.676856342 podStartE2EDuration="1.676856342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:42.674351404 +0000 UTC m=+1810.118662422" watchObservedRunningTime="2026-07-28 20:29:42.676856342 +0000 UTC m=+1810.121167358" Jul 28 20:29:43.327866 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:43.327831 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:43.332841 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:43.332817 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:43.653837 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:43.653758 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:43.654690 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:43.654671 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:29:45.020308 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:45.020269 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:29:55.311368 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:55.311315 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:29:58.655595 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.655561 2581 patch_prober.go:28] interesting pod/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 28 20:29:58.655991 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.655621 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 28 20:29:58.755706 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.755633 2581 generic.go:356] "Generic (PLEG): container finished" podID="861b1170-9496-499c-ad80-aab06e4382a5" containerID="2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142" exitCode=0 Jul 28 20:29:58.755854 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.755705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" event={"ID":"861b1170-9496-499c-ad80-aab06e4382a5","Type":"ContainerDied","Data":"2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142"} Jul 28 20:29:58.756106 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.756088 2581 scope.go:117] "RemoveContainer" containerID="2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142" Jul 28 20:29:58.757105 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.757078 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:29:58.757162 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.757150 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:29:58.757199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:58.757184 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:29:59.763797 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:29:59.763766 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" event={"ID":"861b1170-9496-499c-ad80-aab06e4382a5","Type":"ContainerStarted","Data":"4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e"} Jul 28 20:30:00.133083 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.132995 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5"] Jul 28 20:30:00.135621 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.135593 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb"] Jul 28 20:30:00.135747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.135706 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" Jul 28 20:30:00.137926 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.137906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" Jul 28 20:30:00.141478 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.141456 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:30:00.146300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.146282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" Jul 28 20:30:00.150799 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.150772 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5"] Jul 28 20:30:00.155458 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.155440 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" Jul 28 20:30:00.155928 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.155901 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb"] Jul 28 20:30:00.331347 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.327858 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:30:00.331347 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.327982 2581 patch_prober.go:28] interesting pod/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 28 20:30:00.331347 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.328024 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 28 20:30:00.338112 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.338084 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5"] Jul 28 20:30:00.340891 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:30:00.340834 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfec9c71_c116_4565_bd79_d018a784cc85.slice/crio-994a6b68d7338177431598762e4e88da75c60e03e76326f13bec2c088ee25e2e WatchSource:0}: Error finding container 994a6b68d7338177431598762e4e88da75c60e03e76326f13bec2c088ee25e2e: Status 404 returned error can't find the container with id 994a6b68d7338177431598762e4e88da75c60e03e76326f13bec2c088ee25e2e Jul 28 20:30:00.391553 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.390155 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb"] Jul 28 20:30:00.395289 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:30:00.395263 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9dcd2_60d3_469f_ad06_1acd7c35edbb.slice/crio-4c11a1d99991a0b509eaa75133dfa538f505c693ba010ef8ba136a5122d252ed WatchSource:0}: Error finding container 4c11a1d99991a0b509eaa75133dfa538f505c693ba010ef8ba136a5122d252ed: Status 404 returned error can't find the container with id 4c11a1d99991a0b509eaa75133dfa538f505c693ba010ef8ba136a5122d252ed Jul 28 20:30:00.771488 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.771451 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" event={"ID":"cfec9c71-c116-4565-bd79-d018a784cc85","Type":"ContainerStarted","Data":"8f930f6b5eedf321c8eba8ee9b236968e8c749ec5b8b7cea889c1e89d6def22b"} Jul 28 20:30:00.771901 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.771512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" event={"ID":"cfec9c71-c116-4565-bd79-d018a784cc85","Type":"ContainerStarted","Data":"994a6b68d7338177431598762e4e88da75c60e03e76326f13bec2c088ee25e2e"} Jul 28 20:30:00.772583 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.772565 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" event={"ID":"b7d9dcd2-60d3-469f-ad06-1acd7c35edbb","Type":"ContainerStarted","Data":"4c11a1d99991a0b509eaa75133dfa538f505c693ba010ef8ba136a5122d252ed"} Jul 28 20:30:00.786527 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:00.786473 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" podStartSLOduration=0.786456261 podStartE2EDuration="786.456261ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:00.78563575 +0000 UTC m=+1828.229946766" watchObservedRunningTime="2026-07-28 20:30:00.786456261 +0000 UTC m=+1828.230767285" Jul 28 20:30:01.328428 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:01.328067 2581 patch_prober.go:28] interesting pod/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 28 20:30:01.328428 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:01.328130 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 28 20:30:01.779763 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:01.779724 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" event={"ID":"b7d9dcd2-60d3-469f-ad06-1acd7c35edbb","Type":"ContainerStarted","Data":"a8a8bad24d8b78e0f33ce5dacfa6b9bf82a80c2f6549c973f13fa47c9c751e65"} Jul 28 20:30:01.781238 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:01.781214 2581 generic.go:356] "Generic (PLEG): container finished" podID="cfec9c71-c116-4565-bd79-d018a784cc85" containerID="8f930f6b5eedf321c8eba8ee9b236968e8c749ec5b8b7cea889c1e89d6def22b" exitCode=0 Jul 28 20:30:01.781366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:01.781244 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" event={"ID":"cfec9c71-c116-4565-bd79-d018a784cc85","Type":"ContainerDied","Data":"8f930f6b5eedf321c8eba8ee9b236968e8c749ec5b8b7cea889c1e89d6def22b"} Jul 28 20:30:01.793556 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:01.793496 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" podStartSLOduration=1.7934779 podStartE2EDuration="1.7934779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:01.792789325 +0000 UTC m=+1829.237100340" watchObservedRunningTime="2026-07-28 20:30:01.7934779 +0000 UTC m=+1829.237788917" Jul 28 20:30:02.328283 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:02.328243 2581 patch_prober.go:28] interesting pod/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 28 20:30:02.328522 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:02.328407 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 28 20:30:02.788576 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:02.788543 2581 generic.go:356] "Generic (PLEG): container finished" podID="b7d9dcd2-60d3-469f-ad06-1acd7c35edbb" containerID="a8a8bad24d8b78e0f33ce5dacfa6b9bf82a80c2f6549c973f13fa47c9c751e65" exitCode=0 Jul 28 20:30:02.788964 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:02.788636 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" event={"ID":"b7d9dcd2-60d3-469f-ad06-1acd7c35edbb","Type":"ContainerDied","Data":"a8a8bad24d8b78e0f33ce5dacfa6b9bf82a80c2f6549c973f13fa47c9c751e65"} Jul 28 20:30:02.956600 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:02.956569 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" Jul 28 20:30:03.327827 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:03.327784 2581 patch_prober.go:28] interesting pod/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 28 20:30:03.328077 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:03.328049 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 28 20:30:03.796290 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:03.796249 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" Jul 28 20:30:03.796912 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:03.796886 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754510-vbcd5" event={"ID":"cfec9c71-c116-4565-bd79-d018a784cc85","Type":"ContainerDied","Data":"994a6b68d7338177431598762e4e88da75c60e03e76326f13bec2c088ee25e2e"} Jul 28 20:30:03.796983 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:03.796950 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="994a6b68d7338177431598762e4e88da75c60e03e76326f13bec2c088ee25e2e" Jul 28 20:30:03.929719 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:03.929692 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" Jul 28 20:30:04.331539 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:04.331508 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:30:04.331948 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:04.331925 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:30:04.332571 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:04.332556 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:30:04.802523 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:04.802492 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" Jul 28 20:30:04.802936 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:04.802533 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb" event={"ID":"b7d9dcd2-60d3-469f-ad06-1acd7c35edbb","Type":"ContainerDied","Data":"4c11a1d99991a0b509eaa75133dfa538f505c693ba010ef8ba136a5122d252ed"} Jul 28 20:30:04.802936 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:04.802631 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c11a1d99991a0b509eaa75133dfa538f505c693ba010ef8ba136a5122d252ed" Jul 28 20:30:24.887755 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:24.887709 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb"] Jul 28 20:30:24.889879 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:24.889853 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-89b18cb8-29754510-5vmkb"] Jul 28 20:30:25.152628 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:25.152546 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d9dcd2-60d3-469f-ad06-1acd7c35edbb" path="/var/lib/kubelet/pods/b7d9dcd2-60d3-469f-ad06-1acd7c35edbb/volumes" Jul 28 20:30:26.492561 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:26.492525 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh"] Jul 28 20:30:26.493084 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:26.492737 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" containerID="cri-o://4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e" gracePeriod=30 Jul 28 20:30:29.296605 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:29.296571 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:30:31.740977 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.740950 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:30:31.850816 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.850730 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-credential-socket\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.850952 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.850862 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-istio-token\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.850952 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.850906 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-socket\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.851032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.850951 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g99v\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-kube-api-access-4g99v\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.851032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.850973 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b1170-9496-499c-ad80-aab06e4382a5-istiod-ca-cert\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.851032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.850990 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-data\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.851032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851005 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:30:31.851235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851012 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b1170-9496-499c-ad80-aab06e4382a5-istio-podinfo\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.851235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851067 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-certs\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.851235 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851098 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-envoy\") pod \"861b1170-9496-499c-ad80-aab06e4382a5\" (UID: \"861b1170-9496-499c-ad80-aab06e4382a5\") " Jul 28 20:30:31.851433 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851286 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:30:31.851504 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851472 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:30:31.851577 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851551 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.851636 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851580 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-workload-certs\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.851636 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851596 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-credential-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.851636 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851623 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861b1170-9496-499c-ad80-aab06e4382a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:30:31.851972 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.851952 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:30:31.853213 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.853183 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-kube-api-access-4g99v" (OuterVolumeSpecName: "kube-api-access-4g99v") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "kube-api-access-4g99v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:30:31.853309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.853210 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/861b1170-9496-499c-ad80-aab06e4382a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:30:31.853497 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.853469 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:30:31.853497 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.853472 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "861b1170-9496-499c-ad80-aab06e4382a5" (UID: "861b1170-9496-499c-ad80-aab06e4382a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:30:31.952110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.952078 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g99v\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-kube-api-access-4g99v\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.952110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.952107 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861b1170-9496-499c-ad80-aab06e4382a5-istiod-ca-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.952110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.952117 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-data\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.952386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.952124 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861b1170-9496-499c-ad80-aab06e4382a5-istio-podinfo\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.952386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.952133 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861b1170-9496-499c-ad80-aab06e4382a5-istio-envoy\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.952386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.952142 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861b1170-9496-499c-ad80-aab06e4382a5-istio-token\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:31.958087 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.958056 2581 generic.go:356] "Generic (PLEG): container finished" podID="861b1170-9496-499c-ad80-aab06e4382a5" containerID="4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e" exitCode=0 Jul 28 20:30:31.958210 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.958117 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" Jul 28 20:30:31.958210 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.958143 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" event={"ID":"861b1170-9496-499c-ad80-aab06e4382a5","Type":"ContainerDied","Data":"4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e"} Jul 28 20:30:31.958299 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.958216 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh" event={"ID":"861b1170-9496-499c-ad80-aab06e4382a5","Type":"ContainerDied","Data":"5e30be5baf48efb62d112f3f3d49a0aefdb87879b84f3e568be6fe9cff5a1c12"} Jul 28 20:30:31.958299 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.958237 2581 scope.go:117] "RemoveContainer" containerID="4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e" Jul 28 20:30:31.968021 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.968006 2581 scope.go:117] "RemoveContainer" containerID="2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142" Jul 28 20:30:31.978353 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.978316 2581 scope.go:117] "RemoveContainer" containerID="4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e" Jul 28 20:30:31.978644 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:30:31.978610 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e\": container with ID starting with 4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e not found: ID does not exist" containerID="4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e" Jul 28 20:30:31.978708 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.978652 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e"} err="failed to get container status \"4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e\": rpc error: code = NotFound desc = could not find container \"4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e\": container with ID starting with 4446c8c165ed5ec9b1240d6b703eb2852a2492fd70a477ba74a976ce6feb845e not found: ID does not exist" Jul 28 20:30:31.978708 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.978669 2581 scope.go:117] "RemoveContainer" containerID="2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142" Jul 28 20:30:31.978904 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:30:31.978886 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142\": container with ID starting with 2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142 not found: ID does not exist" containerID="2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142" Jul 28 20:30:31.978948 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.978909 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142"} err="failed to get container status \"2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142\": rpc error: code = NotFound desc = could not find container \"2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142\": container with ID starting with 2e64b197eec8fe4b018e802072c6e73c2af53bb3616a9fcf7af049ca9c987142 not found: ID does not exist" Jul 28 20:30:31.991348 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.991310 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh"] Jul 28 20:30:31.995738 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:31.995717 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-89b18cb8-openshift-default-c85b958f4-bfvsh"] Jul 28 20:30:33.152432 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:33.152398 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="861b1170-9496-499c-ad80-aab06e4382a5" path="/var/lib/kubelet/pods/861b1170-9496-499c-ad80-aab06e4382a5/volumes" Jul 28 20:30:41.507446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:41.507409 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:30:54.422397 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422357 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs"] Jul 28 20:30:54.422782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422716 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" Jul 28 20:30:54.422782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422727 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" Jul 28 20:30:54.422782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422745 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d9dcd2-60d3-469f-ad06-1acd7c35edbb" containerName="cleanup" Jul 28 20:30:54.422782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422751 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d9dcd2-60d3-469f-ad06-1acd7c35edbb" containerName="cleanup" Jul 28 20:30:54.422782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422765 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfec9c71-c116-4565-bd79-d018a784cc85" containerName="cleanup" Jul 28 20:30:54.422782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422771 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfec9c71-c116-4565-bd79-d018a784cc85" containerName="cleanup" Jul 28 20:30:54.422973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422830 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" Jul 28 20:30:54.422973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422838 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d9dcd2-60d3-469f-ad06-1acd7c35edbb" containerName="cleanup" Jul 28 20:30:54.422973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422845 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" Jul 28 20:30:54.422973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.422852 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfec9c71-c116-4565-bd79-d018a784cc85" containerName="cleanup" Jul 28 20:30:54.425933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.425916 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" Jul 28 20:30:54.428249 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.428225 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 20:30:54.428409 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.428389 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:30:54.435557 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.435535 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs"] Jul 28 20:30:54.547140 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.547053 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077ec1e-9a4b-4d5c-acd0-17debc7f894a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs\" (UID: \"e077ec1e-9a4b-4d5c-acd0-17debc7f894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" Jul 28 20:30:54.647720 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.647687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077ec1e-9a4b-4d5c-acd0-17debc7f894a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs\" (UID: \"e077ec1e-9a4b-4d5c-acd0-17debc7f894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" Jul 28 20:30:54.648269 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.648248 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077ec1e-9a4b-4d5c-acd0-17debc7f894a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs\" (UID: \"e077ec1e-9a4b-4d5c-acd0-17debc7f894a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" Jul 28 20:30:54.737291 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.737262 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" Jul 28 20:30:54.872652 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:54.872626 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs"] Jul 28 20:30:54.874621 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:30:54.874592 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode077ec1e_9a4b_4d5c_acd0_17debc7f894a.slice/crio-6b12c8a51078c7975a9775438cf7d9bbfa0f2c2d213279ee4e2911433119def2 WatchSource:0}: Error finding container 6b12c8a51078c7975a9775438cf7d9bbfa0f2c2d213279ee4e2911433119def2: Status 404 returned error can't find the container with id 6b12c8a51078c7975a9775438cf7d9bbfa0f2c2d213279ee4e2911433119def2 Jul 28 20:30:55.085648 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:55.085613 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" event={"ID":"e077ec1e-9a4b-4d5c-acd0-17debc7f894a","Type":"ContainerStarted","Data":"6b12c8a51078c7975a9775438cf7d9bbfa0f2c2d213279ee4e2911433119def2"} Jul 28 20:30:56.091934 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:56.091898 2581 generic.go:356] "Generic (PLEG): container finished" podID="e077ec1e-9a4b-4d5c-acd0-17debc7f894a" containerID="e3becd5a3d8a316363646c25733f2fe0589a21d67603eebcc169733e26af4e21" exitCode=0 Jul 28 20:30:56.092302 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:56.091967 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" event={"ID":"e077ec1e-9a4b-4d5c-acd0-17debc7f894a","Type":"ContainerDied","Data":"e3becd5a3d8a316363646c25733f2fe0589a21d67603eebcc169733e26af4e21"} Jul 28 20:30:57.225415 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:57.225388 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" Jul 28 20:30:57.271838 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:57.271807 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077ec1e-9a4b-4d5c-acd0-17debc7f894a-maas-api-service-ca\") pod \"e077ec1e-9a4b-4d5c-acd0-17debc7f894a\" (UID: \"e077ec1e-9a4b-4d5c-acd0-17debc7f894a\") " Jul 28 20:30:57.272156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:57.272124 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e077ec1e-9a4b-4d5c-acd0-17debc7f894a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e077ec1e-9a4b-4d5c-acd0-17debc7f894a" (UID: "e077ec1e-9a4b-4d5c-acd0-17debc7f894a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:30:57.373075 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:57.372990 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e077ec1e-9a4b-4d5c-acd0-17debc7f894a-maas-api-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:30:58.103670 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:58.103635 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" event={"ID":"e077ec1e-9a4b-4d5c-acd0-17debc7f894a","Type":"ContainerDied","Data":"6b12c8a51078c7975a9775438cf7d9bbfa0f2c2d213279ee4e2911433119def2"} Jul 28 20:30:58.103840 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:58.103684 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b12c8a51078c7975a9775438cf7d9bbfa0f2c2d213279ee4e2911433119def2" Jul 28 20:30:58.103840 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:30:58.103650 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs" Jul 28 20:31:00.985217 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:31:00.985182 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs"] Jul 28 20:31:00.988240 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:31:00.988215 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1bd84f55-qstrs"] Jul 28 20:31:01.153242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:31:01.153210 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e077ec1e-9a4b-4d5c-acd0-17debc7f894a" path="/var/lib/kubelet/pods/e077ec1e-9a4b-4d5c-acd0-17debc7f894a/volumes" Jul 28 20:31:04.791576 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:31:04.791540 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:31:16.411622 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:31:16.411584 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:32:29.292119 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:32:29.292030 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:32:42.492708 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:32:42.492674 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:32:51.507995 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:32:51.507962 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:33:01.135263 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.135231 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn"] Jul 28 20:33:01.135740 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.135663 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" Jul 28 20:33:01.135740 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.135678 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="861b1170-9496-499c-ad80-aab06e4382a5" containerName="istio-proxy" Jul 28 20:33:01.135740 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.135690 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e077ec1e-9a4b-4d5c-acd0-17debc7f894a" containerName="revoke-keys" Jul 28 20:33:01.135740 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.135695 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e077ec1e-9a4b-4d5c-acd0-17debc7f894a" containerName="revoke-keys" Jul 28 20:33:01.135874 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.135766 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e077ec1e-9a4b-4d5c-acd0-17debc7f894a" containerName="revoke-keys" Jul 28 20:33:01.138896 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.138879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.141940 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.141921 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e9e0fd3a-dockercfg-skbcg\"" Jul 28 20:33:01.142041 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.142005 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e9e0fd3a\"" Jul 28 20:33:01.153218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.153192 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn"] Jul 28 20:33:01.190023 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.189988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685396df-0563-4f61-bb78-18cdc54f0383-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.190177 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.190035 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlp5\" (UniqueName: \"kubernetes.io/projected/685396df-0563-4f61-bb78-18cdc54f0383-kube-api-access-5mlp5\") pod \"payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.291261 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.291221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685396df-0563-4f61-bb78-18cdc54f0383-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.291456 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.291281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlp5\" (UniqueName: \"kubernetes.io/projected/685396df-0563-4f61-bb78-18cdc54f0383-kube-api-access-5mlp5\") pod \"payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.291971 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.291951 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685396df-0563-4f61-bb78-18cdc54f0383-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.300070 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.300044 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlp5\" (UniqueName: \"kubernetes.io/projected/685396df-0563-4f61-bb78-18cdc54f0383-kube-api-access-5mlp5\") pod \"payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.449902 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.449821 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:01.591970 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.591939 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn"] Jul 28 20:33:01.593607 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:33:01.593582 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685396df_0563_4f61_bb78_18cdc54f0383.slice/crio-d1aefe7255062bfc40738733c976893179564e2d5f0be356ed8f046b9585fb26 WatchSource:0}: Error finding container d1aefe7255062bfc40738733c976893179564e2d5f0be356ed8f046b9585fb26: Status 404 returned error can't find the container with id d1aefe7255062bfc40738733c976893179564e2d5f0be356ed8f046b9585fb26 Jul 28 20:33:01.595867 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.595851 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:33:01.796189 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:01.796156 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" event={"ID":"685396df-0563-4f61-bb78-18cdc54f0383","Type":"ContainerStarted","Data":"d1aefe7255062bfc40738733c976893179564e2d5f0be356ed8f046b9585fb26"} Jul 28 20:33:03.809587 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:03.809548 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" event={"ID":"685396df-0563-4f61-bb78-18cdc54f0383","Type":"ContainerStarted","Data":"f2b9a69fccc8084fbad963c8570f44103b2235536bd309e575dd749ffb2100fe"} Jul 28 20:33:03.810006 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:03.809675 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:03.827255 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:03.827189 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" podStartSLOduration=1.232620241 podStartE2EDuration="2.827173689s" totalImagesPullingTime="1.594553448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:33:03.825447579 +0000 UTC m=+2011.269758595" watchObservedRunningTime="2026-07-28 20:33:03.827173689 +0000 UTC m=+2011.271484752" Jul 28 20:33:04.985304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:04.985260 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:33:14.818076 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:14.818042 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:32.188456 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.188419 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh"] Jul 28 20:33:32.191784 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.191765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" Jul 28 20:33:32.194037 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.194010 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:33:32.194133 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.194036 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 20:33:32.200714 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.200690 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh"] Jul 28 20:33:32.276882 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.276852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/923ecbde-974d-4537-a61a-f1caea56875d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh\" (UID: \"923ecbde-974d-4537-a61a-f1caea56875d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" Jul 28 20:33:32.377365 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.377309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/923ecbde-974d-4537-a61a-f1caea56875d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh\" (UID: \"923ecbde-974d-4537-a61a-f1caea56875d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" Jul 28 20:33:32.377959 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.377934 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/923ecbde-974d-4537-a61a-f1caea56875d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh\" (UID: \"923ecbde-974d-4537-a61a-f1caea56875d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" Jul 28 20:33:32.503177 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.503145 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" Jul 28 20:33:32.634867 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.634841 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh"] Jul 28 20:33:32.635974 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:33:32.635949 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923ecbde_974d_4537_a61a_f1caea56875d.slice/crio-42961a96a4d0468fef5b1f453c8f1aa95c2013ed8af3eeebea95f31415c5a351 WatchSource:0}: Error finding container 42961a96a4d0468fef5b1f453c8f1aa95c2013ed8af3eeebea95f31415c5a351: Status 404 returned error can't find the container with id 42961a96a4d0468fef5b1f453c8f1aa95c2013ed8af3eeebea95f31415c5a351 Jul 28 20:33:32.976474 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.976441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" event={"ID":"923ecbde-974d-4537-a61a-f1caea56875d","Type":"ContainerStarted","Data":"ed07a3fca8eef1003b71590b96dde8cc7304caf256ae5aa0f56d57614a287388"} Jul 28 20:33:32.976663 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.976498 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" event={"ID":"923ecbde-974d-4537-a61a-f1caea56875d","Type":"ContainerStarted","Data":"42961a96a4d0468fef5b1f453c8f1aa95c2013ed8af3eeebea95f31415c5a351"} Jul 28 20:33:32.994729 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:32.994682 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" podStartSLOduration=0.994666903 podStartE2EDuration="994.666903ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:33:32.993311273 +0000 UTC m=+2040.437622290" watchObservedRunningTime="2026-07-28 20:33:32.994666903 +0000 UTC m=+2040.438977918" Jul 28 20:33:33.983896 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:33.983860 2581 generic.go:356] "Generic (PLEG): container finished" podID="923ecbde-974d-4537-a61a-f1caea56875d" containerID="ed07a3fca8eef1003b71590b96dde8cc7304caf256ae5aa0f56d57614a287388" exitCode=0 Jul 28 20:33:33.984285 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:33.983947 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" event={"ID":"923ecbde-974d-4537-a61a-f1caea56875d","Type":"ContainerDied","Data":"ed07a3fca8eef1003b71590b96dde8cc7304caf256ae5aa0f56d57614a287388"} Jul 28 20:33:35.120018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:35.119990 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" Jul 28 20:33:35.205007 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:35.204978 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/923ecbde-974d-4537-a61a-f1caea56875d-maas-api-service-ca\") pod \"923ecbde-974d-4537-a61a-f1caea56875d\" (UID: \"923ecbde-974d-4537-a61a-f1caea56875d\") " Jul 28 20:33:35.205380 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:35.205358 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/923ecbde-974d-4537-a61a-f1caea56875d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "923ecbde-974d-4537-a61a-f1caea56875d" (UID: "923ecbde-974d-4537-a61a-f1caea56875d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:33:35.306744 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:35.306655 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/923ecbde-974d-4537-a61a-f1caea56875d-maas-api-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:33:35.999181 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:35.999142 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" Jul 28 20:33:35.999181 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:35.999150 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh" event={"ID":"923ecbde-974d-4537-a61a-f1caea56875d","Type":"ContainerDied","Data":"42961a96a4d0468fef5b1f453c8f1aa95c2013ed8af3eeebea95f31415c5a351"} Jul 28 20:33:35.999466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:35.999199 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42961a96a4d0468fef5b1f453c8f1aa95c2013ed8af3eeebea95f31415c5a351" Jul 28 20:33:42.826917 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:42.826884 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn"] Jul 28 20:33:42.827443 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:42.827104 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" podUID="685396df-0563-4f61-bb78-18cdc54f0383" containerName="payload-pre-processing" containerID="cri-o://f2b9a69fccc8084fbad963c8570f44103b2235536bd309e575dd749ffb2100fe" gracePeriod=30 Jul 28 20:33:42.983934 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:42.983886 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh"] Jul 28 20:33:42.986034 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:42.986006 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9e0fd3a-hlwqh"] Jul 28 20:33:43.041939 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.041900 2581 generic.go:356] "Generic (PLEG): container finished" podID="685396df-0563-4f61-bb78-18cdc54f0383" containerID="f2b9a69fccc8084fbad963c8570f44103b2235536bd309e575dd749ffb2100fe" exitCode=0 Jul 28 20:33:43.042115 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.041972 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" event={"ID":"685396df-0563-4f61-bb78-18cdc54f0383","Type":"ContainerDied","Data":"f2b9a69fccc8084fbad963c8570f44103b2235536bd309e575dd749ffb2100fe"} Jul 28 20:33:43.071366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.071343 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:43.152089 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.152007 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="923ecbde-974d-4537-a61a-f1caea56875d" path="/var/lib/kubelet/pods/923ecbde-974d-4537-a61a-f1caea56875d/volumes" Jul 28 20:33:43.179135 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.179102 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685396df-0563-4f61-bb78-18cdc54f0383-plugins-config-volume\") pod \"685396df-0563-4f61-bb78-18cdc54f0383\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " Jul 28 20:33:43.179282 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.179192 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mlp5\" (UniqueName: \"kubernetes.io/projected/685396df-0563-4f61-bb78-18cdc54f0383-kube-api-access-5mlp5\") pod \"685396df-0563-4f61-bb78-18cdc54f0383\" (UID: \"685396df-0563-4f61-bb78-18cdc54f0383\") " Jul 28 20:33:43.179553 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.179534 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685396df-0563-4f61-bb78-18cdc54f0383-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "685396df-0563-4f61-bb78-18cdc54f0383" (UID: "685396df-0563-4f61-bb78-18cdc54f0383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:33:43.181232 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.181201 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685396df-0563-4f61-bb78-18cdc54f0383-kube-api-access-5mlp5" (OuterVolumeSpecName: "kube-api-access-5mlp5") pod "685396df-0563-4f61-bb78-18cdc54f0383" (UID: "685396df-0563-4f61-bb78-18cdc54f0383"). InnerVolumeSpecName "kube-api-access-5mlp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:33:43.279878 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.279843 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685396df-0563-4f61-bb78-18cdc54f0383-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:33:43.279878 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:43.279875 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mlp5\" (UniqueName: \"kubernetes.io/projected/685396df-0563-4f61-bb78-18cdc54f0383-kube-api-access-5mlp5\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:33:44.049472 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:44.049443 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" Jul 28 20:33:44.049922 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:44.049486 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn" event={"ID":"685396df-0563-4f61-bb78-18cdc54f0383","Type":"ContainerDied","Data":"d1aefe7255062bfc40738733c976893179564e2d5f0be356ed8f046b9585fb26"} Jul 28 20:33:44.049922 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:44.049547 2581 scope.go:117] "RemoveContainer" containerID="f2b9a69fccc8084fbad963c8570f44103b2235536bd309e575dd749ffb2100fe" Jul 28 20:33:44.075941 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:44.075911 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn"] Jul 28 20:33:44.082566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:44.082534 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e9e0fd3a-85b4d5545b-l2ffn"] Jul 28 20:33:45.151847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:45.151813 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="685396df-0563-4f61-bb78-18cdc54f0383" path="/var/lib/kubelet/pods/685396df-0563-4f61-bb78-18cdc54f0383/volumes" Jul 28 20:33:45.785831 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:33:45.785800 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:34:13.538605 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.538573 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n"] Jul 28 20:34:13.539043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.538927 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685396df-0563-4f61-bb78-18cdc54f0383" containerName="payload-pre-processing" Jul 28 20:34:13.539043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.538938 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="685396df-0563-4f61-bb78-18cdc54f0383" containerName="payload-pre-processing" Jul 28 20:34:13.539043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.538963 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="923ecbde-974d-4537-a61a-f1caea56875d" containerName="revoke-keys" Jul 28 20:34:13.539043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.538968 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="923ecbde-974d-4537-a61a-f1caea56875d" containerName="revoke-keys" Jul 28 20:34:13.539043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.539021 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="685396df-0563-4f61-bb78-18cdc54f0383" containerName="payload-pre-processing" Jul 28 20:34:13.539043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.539030 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="923ecbde-974d-4537-a61a-f1caea56875d" containerName="revoke-keys" Jul 28 20:34:13.542095 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.542079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.544879 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.544855 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-404fe153\"" Jul 28 20:34:13.545022 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.545004 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-404fe153-dockercfg-cnrzn\"" Jul 28 20:34:13.558262 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.558239 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n"] Jul 28 20:34:13.604566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.604529 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n"] Jul 28 20:34:13.609633 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.609607 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.623538 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.623511 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n"] Jul 28 20:34:13.651908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.651872 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvr6f\" (UniqueName: \"kubernetes.io/projected/40edb17f-029e-49fa-9158-51955d8cb0ec-kube-api-access-hvr6f\") pod \"payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.652071 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.651915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40edb17f-029e-49fa-9158-51955d8cb0ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.752651 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.752615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4763158a-5ea3-43b8-9c88-93f923659b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.752826 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.752659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hcv\" (UniqueName: \"kubernetes.io/projected/4763158a-5ea3-43b8-9c88-93f923659b4a-kube-api-access-99hcv\") pod \"payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.752826 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.752691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr6f\" (UniqueName: \"kubernetes.io/projected/40edb17f-029e-49fa-9158-51955d8cb0ec-kube-api-access-hvr6f\") pod \"payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.752826 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.752726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40edb17f-029e-49fa-9158-51955d8cb0ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.753467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.753449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40edb17f-029e-49fa-9158-51955d8cb0ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.761254 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.761229 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvr6f\" (UniqueName: \"kubernetes.io/projected/40edb17f-029e-49fa-9158-51955d8cb0ec-kube-api-access-hvr6f\") pod \"payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.853068 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.852967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:13.853249 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.853222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4763158a-5ea3-43b8-9c88-93f923659b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.853358 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.853269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hcv\" (UniqueName: \"kubernetes.io/projected/4763158a-5ea3-43b8-9c88-93f923659b4a-kube-api-access-99hcv\") pod \"payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.853949 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.853930 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4763158a-5ea3-43b8-9c88-93f923659b4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.861908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.861886 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hcv\" (UniqueName: \"kubernetes.io/projected/4763158a-5ea3-43b8-9c88-93f923659b4a-kube-api-access-99hcv\") pod \"payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.922053 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.922024 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:13.993558 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:13.993522 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n"] Jul 28 20:34:13.995466 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:34:13.995436 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40edb17f_029e_49fa_9158_51955d8cb0ec.slice/crio-44c2d0c575748e83fa8e3985de3c86a159e8af56b5bd39a6dad2eac0884d66d2 WatchSource:0}: Error finding container 44c2d0c575748e83fa8e3985de3c86a159e8af56b5bd39a6dad2eac0884d66d2: Status 404 returned error can't find the container with id 44c2d0c575748e83fa8e3985de3c86a159e8af56b5bd39a6dad2eac0884d66d2 Jul 28 20:34:14.068838 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.068813 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n"] Jul 28 20:34:14.070608 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:34:14.070582 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4763158a_5ea3_43b8_9c88_93f923659b4a.slice/crio-359f02b2741c6b7b4748391e89c6ff2473151dc011a3f50abd13fc59ef279c7b WatchSource:0}: Error finding container 359f02b2741c6b7b4748391e89c6ff2473151dc011a3f50abd13fc59ef279c7b: Status 404 returned error can't find the container with id 359f02b2741c6b7b4748391e89c6ff2473151dc011a3f50abd13fc59ef279c7b Jul 28 20:34:14.226736 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.226701 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" event={"ID":"4763158a-5ea3-43b8-9c88-93f923659b4a","Type":"ContainerStarted","Data":"7eae71a6b3d368a666a8e96a2122aebb589721bbb3d60a95b0472b694b5d55c7"} Jul 28 20:34:14.226932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.226754 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" event={"ID":"4763158a-5ea3-43b8-9c88-93f923659b4a","Type":"ContainerStarted","Data":"359f02b2741c6b7b4748391e89c6ff2473151dc011a3f50abd13fc59ef279c7b"} Jul 28 20:34:14.226932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.226812 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:14.228659 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.228632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" event={"ID":"40edb17f-029e-49fa-9158-51955d8cb0ec","Type":"ContainerStarted","Data":"20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3"} Jul 28 20:34:14.228788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.228681 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" event={"ID":"40edb17f-029e-49fa-9158-51955d8cb0ec","Type":"ContainerStarted","Data":"44c2d0c575748e83fa8e3985de3c86a159e8af56b5bd39a6dad2eac0884d66d2"} Jul 28 20:34:14.228788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.228707 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:14.250030 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.249978 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" podStartSLOduration=1.249962322 podStartE2EDuration="1.249962322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:34:14.24699913 +0000 UTC m=+2081.691310145" watchObservedRunningTime="2026-07-28 20:34:14.249962322 +0000 UTC m=+2081.694273343" Jul 28 20:34:14.267351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:14.267286 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" podStartSLOduration=1.267266549 podStartE2EDuration="1.267266549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:34:14.26473085 +0000 UTC m=+2081.709041867" watchObservedRunningTime="2026-07-28 20:34:14.267266549 +0000 UTC m=+2081.711577639" Jul 28 20:34:17.594911 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:17.594876 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:34:25.236501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:25.236466 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:34:25.236953 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:25.236540 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:34:57.595070 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.595037 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq"] Jul 28 20:34:57.598719 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.598699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.600734 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.600710 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-404fe153\"/\"kube-root-ca.crt\"" Jul 28 20:34:57.601564 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.601542 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-404fe153\"/\"e2e-lifecycle-model-404fe153-kserve-self-signed-certs\"" Jul 28 20:34:57.601564 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.601556 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-404fe153\"/\"default-dockercfg-8hlxr\"" Jul 28 20:34:57.601723 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.601594 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-404fe153\"/\"openshift-service-ca.crt\"" Jul 28 20:34:57.608101 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.608078 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq"] Jul 28 20:34:57.741206 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.741169 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tmp-dir\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.741424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.741238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-model-cache\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.741424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.741267 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tls-certs\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.741424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.741314 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-dshm\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.741424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.741410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-home\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.741669 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.741454 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkz4z\" (UniqueName: \"kubernetes.io/projected/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-kube-api-access-gkz4z\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.842754 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.842714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-dshm\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.842932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.842769 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-home\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.842932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.842800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkz4z\" (UniqueName: \"kubernetes.io/projected/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-kube-api-access-gkz4z\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.842932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.842839 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tmp-dir\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.842932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.842906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-model-cache\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.843129 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.842941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tls-certs\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.843191 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.843127 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-home\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.843276 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.843256 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tmp-dir\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.843372 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.843304 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-model-cache\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.845153 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.845093 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-dshm\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.845349 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.845313 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tls-certs\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.851391 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.851364 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkz4z\" (UniqueName: \"kubernetes.io/projected/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-kube-api-access-gkz4z\") pod \"e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:57.909247 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:57.909204 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:34:58.048998 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:58.048968 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq"] Jul 28 20:34:58.049993 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:34:58.049969 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62ee52dc_f9a9_4b21_8b1f_c333b81f70da.slice/crio-5c21f1de5b59b1bc43647444dae13cbe79c52f44a825fcdb2fd592964c0a4b8d WatchSource:0}: Error finding container 5c21f1de5b59b1bc43647444dae13cbe79c52f44a825fcdb2fd592964c0a4b8d: Status 404 returned error can't find the container with id 5c21f1de5b59b1bc43647444dae13cbe79c52f44a825fcdb2fd592964c0a4b8d Jul 28 20:34:58.479110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:58.479066 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" event={"ID":"62ee52dc-f9a9-4b21-8b1f-c333b81f70da","Type":"ContainerStarted","Data":"5c21f1de5b59b1bc43647444dae13cbe79c52f44a825fcdb2fd592964c0a4b8d"} Jul 28 20:34:59.485890 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:59.485856 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" event={"ID":"62ee52dc-f9a9-4b21-8b1f-c333b81f70da","Type":"ContainerStarted","Data":"1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d"} Jul 28 20:34:59.505934 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:34:59.505882 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" podStartSLOduration=1.470267968 podStartE2EDuration="2.505863936s" totalImagesPullingTime="1.035595968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:34:59.504100706 +0000 UTC m=+2126.948411722" watchObservedRunningTime="2026-07-28 20:34:59.505863936 +0000 UTC m=+2126.950174952" Jul 28 20:35:00.979408 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:00.979370 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:35:07.909457 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:07.909415 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:35:07.909858 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:07.909478 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:35:07.922411 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:07.922385 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:35:08.547790 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:08.547755 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:35:20.294135 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:20.294101 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:35:33.237828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:33.237754 2581 scope.go:117] "RemoveContainer" containerID="29994152a017f98af03ef0b32520b7c886488b887809ddf14570e85c2c88b176" Jul 28 20:35:41.882715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:41.882683 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:35:43.468877 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.468843 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n"] Jul 28 20:35:43.476089 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.476058 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n"] Jul 28 20:35:43.476366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.476304 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" podUID="4763158a-5ea3-43b8-9c88-93f923659b4a" containerName="payload-pre-processing" containerID="cri-o://7eae71a6b3d368a666a8e96a2122aebb589721bbb3d60a95b0472b694b5d55c7" gracePeriod=30 Jul 28 20:35:43.740103 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.740068 2581 generic.go:356] "Generic (PLEG): container finished" podID="4763158a-5ea3-43b8-9c88-93f923659b4a" containerID="7eae71a6b3d368a666a8e96a2122aebb589721bbb3d60a95b0472b694b5d55c7" exitCode=0 Jul 28 20:35:43.740251 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.740142 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" event={"ID":"4763158a-5ea3-43b8-9c88-93f923659b4a","Type":"ContainerDied","Data":"7eae71a6b3d368a666a8e96a2122aebb589721bbb3d60a95b0472b694b5d55c7"} Jul 28 20:35:43.740251 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.740200 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" event={"ID":"4763158a-5ea3-43b8-9c88-93f923659b4a","Type":"ContainerDied","Data":"359f02b2741c6b7b4748391e89c6ff2473151dc011a3f50abd13fc59ef279c7b"} Jul 28 20:35:43.740251 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.740224 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="359f02b2741c6b7b4748391e89c6ff2473151dc011a3f50abd13fc59ef279c7b" Jul 28 20:35:43.740384 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.740361 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" podUID="40edb17f-029e-49fa-9158-51955d8cb0ec" containerName="payload-processing" containerID="cri-o://20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3" gracePeriod=30 Jul 28 20:35:43.741728 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.741711 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:35:43.871926 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.871894 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99hcv\" (UniqueName: \"kubernetes.io/projected/4763158a-5ea3-43b8-9c88-93f923659b4a-kube-api-access-99hcv\") pod \"4763158a-5ea3-43b8-9c88-93f923659b4a\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " Jul 28 20:35:43.872111 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.872015 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4763158a-5ea3-43b8-9c88-93f923659b4a-plugins-config-volume\") pod \"4763158a-5ea3-43b8-9c88-93f923659b4a\" (UID: \"4763158a-5ea3-43b8-9c88-93f923659b4a\") " Jul 28 20:35:43.872819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.872786 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4763158a-5ea3-43b8-9c88-93f923659b4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4763158a-5ea3-43b8-9c88-93f923659b4a" (UID: "4763158a-5ea3-43b8-9c88-93f923659b4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:35:43.875453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.873229 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4763158a-5ea3-43b8-9c88-93f923659b4a-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:35:43.881070 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.881036 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4763158a-5ea3-43b8-9c88-93f923659b4a-kube-api-access-99hcv" (OuterVolumeSpecName: "kube-api-access-99hcv") pod "4763158a-5ea3-43b8-9c88-93f923659b4a" (UID: "4763158a-5ea3-43b8-9c88-93f923659b4a"). InnerVolumeSpecName "kube-api-access-99hcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:35:43.974305 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.974221 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99hcv\" (UniqueName: \"kubernetes.io/projected/4763158a-5ea3-43b8-9c88-93f923659b4a-kube-api-access-99hcv\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:35:43.984616 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:43.984588 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:35:44.075333 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.075286 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40edb17f-029e-49fa-9158-51955d8cb0ec-plugins-config-volume\") pod \"40edb17f-029e-49fa-9158-51955d8cb0ec\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " Jul 28 20:35:44.075538 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.075359 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvr6f\" (UniqueName: \"kubernetes.io/projected/40edb17f-029e-49fa-9158-51955d8cb0ec-kube-api-access-hvr6f\") pod \"40edb17f-029e-49fa-9158-51955d8cb0ec\" (UID: \"40edb17f-029e-49fa-9158-51955d8cb0ec\") " Jul 28 20:35:44.075766 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.075739 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40edb17f-029e-49fa-9158-51955d8cb0ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40edb17f-029e-49fa-9158-51955d8cb0ec" (UID: "40edb17f-029e-49fa-9158-51955d8cb0ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:35:44.077451 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.077424 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40edb17f-029e-49fa-9158-51955d8cb0ec-kube-api-access-hvr6f" (OuterVolumeSpecName: "kube-api-access-hvr6f") pod "40edb17f-029e-49fa-9158-51955d8cb0ec" (UID: "40edb17f-029e-49fa-9158-51955d8cb0ec"). InnerVolumeSpecName "kube-api-access-hvr6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:35:44.176950 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.176909 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvr6f\" (UniqueName: \"kubernetes.io/projected/40edb17f-029e-49fa-9158-51955d8cb0ec-kube-api-access-hvr6f\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:35:44.176950 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.176942 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40edb17f-029e-49fa-9158-51955d8cb0ec-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:35:44.747021 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.746929 2581 generic.go:356] "Generic (PLEG): container finished" podID="40edb17f-029e-49fa-9158-51955d8cb0ec" containerID="20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3" exitCode=0 Jul 28 20:35:44.747021 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.746968 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" Jul 28 20:35:44.747502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.747015 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" event={"ID":"40edb17f-029e-49fa-9158-51955d8cb0ec","Type":"ContainerDied","Data":"20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3"} Jul 28 20:35:44.747502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.747068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n" event={"ID":"40edb17f-029e-49fa-9158-51955d8cb0ec","Type":"ContainerDied","Data":"44c2d0c575748e83fa8e3985de3c86a159e8af56b5bd39a6dad2eac0884d66d2"} Jul 28 20:35:44.747502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.747098 2581 scope.go:117] "RemoveContainer" containerID="20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3" Jul 28 20:35:44.747502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.747155 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n" Jul 28 20:35:44.756414 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.756346 2581 scope.go:117] "RemoveContainer" containerID="20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3" Jul 28 20:35:44.756675 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:35:44.756654 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3\": container with ID starting with 20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3 not found: ID does not exist" containerID="20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3" Jul 28 20:35:44.756741 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.756683 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3"} err="failed to get container status \"20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3\": rpc error: code = NotFound desc = could not find container \"20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3\": container with ID starting with 20018d60bbee74ef15f772e4495921ed487769578d510adca801691a87ddd7a3 not found: ID does not exist" Jul 28 20:35:44.771486 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.771452 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n"] Jul 28 20:35:44.773735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.773712 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-404fe153-fdf8755db-gbl5n"] Jul 28 20:35:44.782929 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.782903 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n"] Jul 28 20:35:44.787001 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:44.786981 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-404fe153-84c446dd88-hln4n"] Jul 28 20:35:45.155146 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:45.155067 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40edb17f-029e-49fa-9158-51955d8cb0ec" path="/var/lib/kubelet/pods/40edb17f-029e-49fa-9158-51955d8cb0ec/volumes" Jul 28 20:35:45.155451 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:45.155438 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4763158a-5ea3-43b8-9c88-93f923659b4a" path="/var/lib/kubelet/pods/4763158a-5ea3-43b8-9c88-93f923659b4a/volumes" Jul 28 20:35:51.978358 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:51.978304 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:35:52.610750 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:35:52.610720 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq"] Jul 28 20:36:07.630340 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.630276 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" podUID="62ee52dc-f9a9-4b21-8b1f-c333b81f70da" containerName="main" containerID="cri-o://1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d" gracePeriod=45 Jul 28 20:36:07.879061 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.879032 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:36:07.881735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.881672 2581 generic.go:356] "Generic (PLEG): container finished" podID="62ee52dc-f9a9-4b21-8b1f-c333b81f70da" containerID="1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d" exitCode=0 Jul 28 20:36:07.881735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.881700 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" Jul 28 20:36:07.881735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.881723 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" event={"ID":"62ee52dc-f9a9-4b21-8b1f-c333b81f70da","Type":"ContainerDied","Data":"1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d"} Jul 28 20:36:07.881895 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.881766 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq" event={"ID":"62ee52dc-f9a9-4b21-8b1f-c333b81f70da","Type":"ContainerDied","Data":"5c21f1de5b59b1bc43647444dae13cbe79c52f44a825fcdb2fd592964c0a4b8d"} Jul 28 20:36:07.881895 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.881783 2581 scope.go:117] "RemoveContainer" containerID="1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d" Jul 28 20:36:07.890455 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.890439 2581 scope.go:117] "RemoveContainer" containerID="1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d" Jul 28 20:36:07.890740 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:36:07.890720 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d\": container with ID starting with 1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d not found: ID does not exist" containerID="1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d" Jul 28 20:36:07.890800 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.890753 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d"} err="failed to get container status \"1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d\": rpc error: code = NotFound desc = could not find container \"1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d\": container with ID starting with 1d16a565983c8789cb5a19bba10b46fabebb52541e609a190f34661d9d44d52d not found: ID does not exist" Jul 28 20:36:07.897816 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.897795 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-model-cache\") pod \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " Jul 28 20:36:07.897933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.897834 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkz4z\" (UniqueName: \"kubernetes.io/projected/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-kube-api-access-gkz4z\") pod \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " Jul 28 20:36:07.897933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.897879 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tls-certs\") pod \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " Jul 28 20:36:07.897933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.897929 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tmp-dir\") pod \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " Jul 28 20:36:07.898091 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.898032 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-dshm\") pod \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " Jul 28 20:36:07.898091 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.898078 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-model-cache" (OuterVolumeSpecName: "model-cache") pod "62ee52dc-f9a9-4b21-8b1f-c333b81f70da" (UID: "62ee52dc-f9a9-4b21-8b1f-c333b81f70da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:36:07.898196 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.898092 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-home\") pod \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\" (UID: \"62ee52dc-f9a9-4b21-8b1f-c333b81f70da\") " Jul 28 20:36:07.898466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.898277 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62ee52dc-f9a9-4b21-8b1f-c333b81f70da" (UID: "62ee52dc-f9a9-4b21-8b1f-c333b81f70da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:36:07.898466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.898406 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-model-cache\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:36:07.898639 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.898581 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-home" (OuterVolumeSpecName: "home") pod "62ee52dc-f9a9-4b21-8b1f-c333b81f70da" (UID: "62ee52dc-f9a9-4b21-8b1f-c333b81f70da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:36:07.901139 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.901111 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-kube-api-access-gkz4z" (OuterVolumeSpecName: "kube-api-access-gkz4z") pod "62ee52dc-f9a9-4b21-8b1f-c333b81f70da" (UID: "62ee52dc-f9a9-4b21-8b1f-c333b81f70da"). InnerVolumeSpecName "kube-api-access-gkz4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:36:07.901391 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.901355 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-dshm" (OuterVolumeSpecName: "dshm") pod "62ee52dc-f9a9-4b21-8b1f-c333b81f70da" (UID: "62ee52dc-f9a9-4b21-8b1f-c333b81f70da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:36:07.901490 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.901403 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62ee52dc-f9a9-4b21-8b1f-c333b81f70da" (UID: "62ee52dc-f9a9-4b21-8b1f-c333b81f70da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 20:36:07.999111 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.999073 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tls-certs\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:36:07.999111 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.999100 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-tmp-dir\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:36:07.999111 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.999110 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-dshm\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:36:07.999111 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.999118 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-home\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:36:07.999397 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:07.999128 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkz4z\" (UniqueName: \"kubernetes.io/projected/62ee52dc-f9a9-4b21-8b1f-c333b81f70da-kube-api-access-gkz4z\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:36:08.203204 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:08.203176 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq"] Jul 28 20:36:08.206247 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:08.206221 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-404fe153/e2e-lifecycle-model-404fe153-kserve-8477bd79ff-x65xq"] Jul 28 20:36:09.158954 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:09.158911 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ee52dc-f9a9-4b21-8b1f-c333b81f70da" path="/var/lib/kubelet/pods/62ee52dc-f9a9-4b21-8b1f-c333b81f70da/volumes" Jul 28 20:36:33.268227 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:33.268188 2581 scope.go:117] "RemoveContainer" containerID="a8a8bad24d8b78e0f33ce5dacfa6b9bf82a80c2f6549c973f13fa47c9c751e65" Jul 28 20:36:33.277769 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:33.277748 2581 scope.go:117] "RemoveContainer" containerID="a201595e37157ae4c866685c75f8e7ea11b3d4155088d8731b8b3845d0ece7ba" Jul 28 20:36:39.784409 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:39.784370 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:36:51.682224 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:36:51.682175 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:37:33.322084 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:33.322043 2581 scope.go:117] "RemoveContainer" containerID="e3becd5a3d8a316363646c25733f2fe0589a21d67603eebcc169733e26af4e21" Jul 28 20:37:36.918860 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.918824 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd"] Jul 28 20:37:36.919260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919192 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40edb17f-029e-49fa-9158-51955d8cb0ec" containerName="payload-processing" Jul 28 20:37:36.919260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919203 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40edb17f-029e-49fa-9158-51955d8cb0ec" containerName="payload-processing" Jul 28 20:37:36.919260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919211 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ee52dc-f9a9-4b21-8b1f-c333b81f70da" containerName="main" Jul 28 20:37:36.919260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919216 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ee52dc-f9a9-4b21-8b1f-c333b81f70da" containerName="main" Jul 28 20:37:36.919260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919224 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763158a-5ea3-43b8-9c88-93f923659b4a" containerName="payload-pre-processing" Jul 28 20:37:36.919260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919230 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763158a-5ea3-43b8-9c88-93f923659b4a" containerName="payload-pre-processing" Jul 28 20:37:36.919510 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919302 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="40edb17f-029e-49fa-9158-51955d8cb0ec" containerName="payload-processing" Jul 28 20:37:36.919510 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919311 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4763158a-5ea3-43b8-9c88-93f923659b4a" containerName="payload-pre-processing" Jul 28 20:37:36.919510 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.919335 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ee52dc-f9a9-4b21-8b1f-c333b81f70da" containerName="main" Jul 28 20:37:36.923603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.923587 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:36.925920 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.925897 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7ff6b3-dockercfg-wks4t\"" Jul 28 20:37:36.926192 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.926175 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7ff6b3\"" Jul 28 20:37:36.932889 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.932867 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd"] Jul 28 20:37:36.981013 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.980978 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg"] Jul 28 20:37:36.985207 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.985183 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:36.996518 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.996483 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mckgw\" (UniqueName: \"kubernetes.io/projected/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-kube-api-access-mckgw\") pod \"payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:36.996677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.996554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tfwk\" (UniqueName: \"kubernetes.io/projected/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-kube-api-access-7tfwk\") pod \"payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:36.996677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.996622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:36.996677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.996672 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:36.999153 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:36.999132 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg"] Jul 28 20:37:37.097279 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.097245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tfwk\" (UniqueName: \"kubernetes.io/projected/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-kube-api-access-7tfwk\") pod \"payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:37.097492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.097297 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:37.097492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.097361 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:37.097492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.097409 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mckgw\" (UniqueName: \"kubernetes.io/projected/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-kube-api-access-mckgw\") pod \"payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:37.098084 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.098063 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:37.098127 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.098077 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:37.105613 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.105590 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tfwk\" (UniqueName: \"kubernetes.io/projected/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-kube-api-access-7tfwk\") pod \"payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:37.106054 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.106034 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mckgw\" (UniqueName: \"kubernetes.io/projected/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-kube-api-access-mckgw\") pod \"payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:37.235056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.235025 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:37.298266 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.298182 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:37.380926 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.380455 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd"] Jul 28 20:37:37.392303 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.392184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" event={"ID":"d9a81bb1-a643-4e3f-bd55-0e85007f4a80","Type":"ContainerStarted","Data":"2fc5d4363fff35e11943123a6aeecf43cb629c8a83373f75463ac88b7f9a67cb"} Jul 28 20:37:37.459815 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:37.459790 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg"] Jul 28 20:37:37.460996 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:37:37.460970 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfa55f4_53f2_4a11_81de_d3de1ebbf8a4.slice/crio-eebf9b4b24437716fa034b7d6cb49b835474931aba7651c0509f26f958483b3a WatchSource:0}: Error finding container eebf9b4b24437716fa034b7d6cb49b835474931aba7651c0509f26f958483b3a: Status 404 returned error can't find the container with id eebf9b4b24437716fa034b7d6cb49b835474931aba7651c0509f26f958483b3a Jul 28 20:37:38.400259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:38.400218 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" event={"ID":"d9a81bb1-a643-4e3f-bd55-0e85007f4a80","Type":"ContainerStarted","Data":"42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2"} Jul 28 20:37:38.400743 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:38.400306 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:37:38.401778 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:38.401751 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" event={"ID":"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4","Type":"ContainerStarted","Data":"89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9"} Jul 28 20:37:38.401918 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:38.401796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" event={"ID":"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4","Type":"ContainerStarted","Data":"eebf9b4b24437716fa034b7d6cb49b835474931aba7651c0509f26f958483b3a"} Jul 28 20:37:38.401918 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:38.401840 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:38.437859 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:38.437807 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" podStartSLOduration=2.437792574 podStartE2EDuration="2.437792574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:37:38.435965525 +0000 UTC m=+2285.880276543" watchObservedRunningTime="2026-07-28 20:37:38.437792574 +0000 UTC m=+2285.882103567" Jul 28 20:37:38.467154 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:38.467084 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" podStartSLOduration=2.467062728 podStartE2EDuration="2.467062728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:37:38.46660521 +0000 UTC m=+2285.910916218" watchObservedRunningTime="2026-07-28 20:37:38.467062728 +0000 UTC m=+2285.911373745" Jul 28 20:37:41.584334 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:41.584285 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:37:49.411039 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:49.411003 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:37:49.411607 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:37:49.411308 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:38:01.022817 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.022783 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p"] Jul 28 20:38:01.026822 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.026799 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.029710 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.029692 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d38c0f-openshift-default-dockercfg-pgddd\"" Jul 28 20:38:01.040640 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.040619 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p"] Jul 28 20:38:01.113115 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlx2\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-kube-api-access-rtlx2\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113276 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113138 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-socket\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113276 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-certs\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-envoy\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113487 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-credential-socket\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113487 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-data\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113594 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-istio-token\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113594 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113553 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d8cc1c-5768-48de-9963-1d63a6516887-istio-podinfo\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.113594 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.113579 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d8cc1c-5768-48de-9963-1d63a6516887-istiod-ca-cert\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.214614 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-socket\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.214793 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-certs\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.214793 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-envoy\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.214793 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-credential-socket\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.214793 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-data\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.214978 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214878 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-istio-token\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.214978 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214946 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d8cc1c-5768-48de-9963-1d63a6516887-istio-podinfo\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.215081 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.214977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d8cc1c-5768-48de-9963-1d63a6516887-istiod-ca-cert\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.215081 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.215023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-socket\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.215081 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.215055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlx2\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-kube-api-access-rtlx2\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.215081 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.215062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-certs\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.215287 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.215085 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-data\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.215287 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.215138 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-credential-socket\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.215634 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.215614 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d8cc1c-5768-48de-9963-1d63a6516887-istiod-ca-cert\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.217174 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.217148 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-envoy\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.217275 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.217255 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d8cc1c-5768-48de-9963-1d63a6516887-istio-podinfo\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.223297 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.223278 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlx2\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-kube-api-access-rtlx2\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.223460 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.223443 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-istio-token\") pod \"e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.337845 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.337752 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:01.474650 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.474625 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p"] Jul 28 20:38:01.476077 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:38:01.476055 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d8cc1c_5768_48de_9963_1d63a6516887.slice/crio-abf7b55730bbd7b3b094be83e45570a3d9748eb72003c3f2f4aaea06936f7682 WatchSource:0}: Error finding container abf7b55730bbd7b3b094be83e45570a3d9748eb72003c3f2f4aaea06936f7682: Status 404 returned error can't find the container with id abf7b55730bbd7b3b094be83e45570a3d9748eb72003c3f2f4aaea06936f7682 Jul 28 20:38:01.478378 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.478349 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:38:01.478443 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.478423 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:38:01.478481 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.478451 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:38:01.534219 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:01.534191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" event={"ID":"33d8cc1c-5768-48de-9963-1d63a6516887","Type":"ContainerStarted","Data":"abf7b55730bbd7b3b094be83e45570a3d9748eb72003c3f2f4aaea06936f7682"} Jul 28 20:38:02.541372 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:02.541334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" event={"ID":"33d8cc1c-5768-48de-9963-1d63a6516887","Type":"ContainerStarted","Data":"c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90"} Jul 28 20:38:02.562628 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:02.562573 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podStartSLOduration=1.562558121 podStartE2EDuration="1.562558121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:02.56019495 +0000 UTC m=+2310.004505966" watchObservedRunningTime="2026-07-28 20:38:02.562558121 +0000 UTC m=+2310.006869175" Jul 28 20:38:03.338780 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:03.338746 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:03.343843 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:03.343820 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:03.546956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:03.546919 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:03.547798 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:03.547780 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:05.428444 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.428408 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx"] Jul 28 20:38:05.432038 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.432018 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.434852 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.434833 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d38c0f\"" Jul 28 20:38:05.448971 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.448948 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx"] Jul 28 20:38:05.499394 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.499361 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr"] Jul 28 20:38:05.503087 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.503070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.507033 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.507015 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d38c0f-dockercfg-89s58\"" Jul 28 20:38:05.525225 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.525204 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr"] Jul 28 20:38:05.556122 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.556081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.556304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.556177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245fl\" (UniqueName: \"kubernetes.io/projected/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-kube-api-access-245fl\") pod \"payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.657010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.656974 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggq8g\" (UniqueName: \"kubernetes.io/projected/90978197-8f8b-405d-9fb5-1270ebc1ecea-kube-api-access-ggq8g\") pod \"payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.657218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.657036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90978197-8f8b-405d-9fb5-1270ebc1ecea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.657218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.657086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.657218 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.657133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245fl\" (UniqueName: \"kubernetes.io/projected/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-kube-api-access-245fl\") pod \"payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.657784 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.657763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.666234 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.666210 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-245fl\" (UniqueName: \"kubernetes.io/projected/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-kube-api-access-245fl\") pod \"payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.742846 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.742811 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:05.758438 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.758370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggq8g\" (UniqueName: \"kubernetes.io/projected/90978197-8f8b-405d-9fb5-1270ebc1ecea-kube-api-access-ggq8g\") pod \"payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.758590 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.758470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90978197-8f8b-405d-9fb5-1270ebc1ecea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.759223 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.759201 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90978197-8f8b-405d-9fb5-1270ebc1ecea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.767386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.767363 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggq8g\" (UniqueName: \"kubernetes.io/projected/90978197-8f8b-405d-9fb5-1270ebc1ecea-kube-api-access-ggq8g\") pod \"payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.813055 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.813023 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:05.897851 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.897820 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx"] Jul 28 20:38:05.897995 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:38:05.897971 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce01018_f26b_4ca7_bcbd_f08903df8d4e.slice/crio-63c5e16647db529820722f28c346b23dbb8ba256d62d4eb0ae8c756f4b1e8ce9 WatchSource:0}: Error finding container 63c5e16647db529820722f28c346b23dbb8ba256d62d4eb0ae8c756f4b1e8ce9: Status 404 returned error can't find the container with id 63c5e16647db529820722f28c346b23dbb8ba256d62d4eb0ae8c756f4b1e8ce9 Jul 28 20:38:05.973578 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:05.973553 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr"] Jul 28 20:38:05.974847 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:38:05.974824 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90978197_8f8b_405d_9fb5_1270ebc1ecea.slice/crio-41a8e8a41eec5f714c4d30e16f1457f2575f822a28427b6a6f4088ae3b1fc93e WatchSource:0}: Error finding container 41a8e8a41eec5f714c4d30e16f1457f2575f822a28427b6a6f4088ae3b1fc93e: Status 404 returned error can't find the container with id 41a8e8a41eec5f714c4d30e16f1457f2575f822a28427b6a6f4088ae3b1fc93e Jul 28 20:38:06.568681 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.568640 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" event={"ID":"90978197-8f8b-405d-9fb5-1270ebc1ecea","Type":"ContainerStarted","Data":"63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10"} Jul 28 20:38:06.569115 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.568747 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:06.569115 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.568777 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" event={"ID":"90978197-8f8b-405d-9fb5-1270ebc1ecea","Type":"ContainerStarted","Data":"41a8e8a41eec5f714c4d30e16f1457f2575f822a28427b6a6f4088ae3b1fc93e"} Jul 28 20:38:06.570270 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.570241 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" event={"ID":"1ce01018-f26b-4ca7-bcbd-f08903df8d4e","Type":"ContainerStarted","Data":"154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22"} Jul 28 20:38:06.570415 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.570285 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" event={"ID":"1ce01018-f26b-4ca7-bcbd-f08903df8d4e","Type":"ContainerStarted","Data":"63c5e16647db529820722f28c346b23dbb8ba256d62d4eb0ae8c756f4b1e8ce9"} Jul 28 20:38:06.570415 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.570304 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:06.610691 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.610637 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" podStartSLOduration=1.610623895 podStartE2EDuration="1.610623895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:06.609336277 +0000 UTC m=+2314.053647289" watchObservedRunningTime="2026-07-28 20:38:06.610623895 +0000 UTC m=+2314.054934889" Jul 28 20:38:06.654824 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:06.654732 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" podStartSLOduration=1.6547183159999999 podStartE2EDuration="1.654718316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:06.652040139 +0000 UTC m=+2314.096351155" watchObservedRunningTime="2026-07-28 20:38:06.654718316 +0000 UTC m=+2314.099029371" Jul 28 20:38:11.882366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:11.882313 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:38:17.578817 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:17.578787 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:38:17.579309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:17.578863 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:38:19.340306 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.339506 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:36158->10.134.0.57:15021: read: connection reset by peer" start-of-body= Jul 28 20:38:19.340306 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.339580 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:36158->10.134.0.57:15021: read: connection reset by peer" Jul 28 20:38:19.656954 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.656857 2581 generic.go:356] "Generic (PLEG): container finished" podID="33d8cc1c-5768-48de-9963-1d63a6516887" containerID="c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90" exitCode=0 Jul 28 20:38:19.656954 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.656930 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" event={"ID":"33d8cc1c-5768-48de-9963-1d63a6516887","Type":"ContainerDied","Data":"c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90"} Jul 28 20:38:19.657352 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.657338 2581 scope.go:117] "RemoveContainer" containerID="c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90" Jul 28 20:38:19.658392 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.658368 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:38:19.658449 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.658430 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:38:19.658488 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:19.658460 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:38:20.667005 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:20.666973 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" event={"ID":"33d8cc1c-5768-48de-9963-1d63a6516887","Type":"ContainerStarted","Data":"c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30"} Jul 28 20:38:21.337891 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:21.337860 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:21.338122 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:21.338099 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:21.338177 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:21.338148 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:22.338524 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:22.338481 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:22.338907 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:22.338538 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:23.338754 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:23.338714 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:23.339210 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:23.338773 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:24.338637 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:24.338601 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:24.338821 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:24.338654 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:25.338084 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:25.338051 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:25.338257 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:25.338106 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:26.339173 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:26.339132 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:26.339595 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:26.339190 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:27.339089 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:27.339055 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:27.339272 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:27.339108 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:28.339029 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:28.338988 2581 patch_prober.go:28] interesting pod/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 20:38:28.339237 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:28.339051 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 20:38:29.342528 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:29.342488 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:29.342995 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:29.342890 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:29.343525 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:29.343504 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:38:43.879445 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:43.879413 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:38:57.902095 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:38:57.902061 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:39:33.352071 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:39:33.352042 2581 scope.go:117] "RemoveContainer" containerID="ed07a3fca8eef1003b71590b96dde8cc7304caf256ae5aa0f56d57614a287388" Jul 28 20:39:51.786827 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:39:51.786792 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:40:06.706371 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:06.706262 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:40:33.372580 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:33.372546 2581 scope.go:117] "RemoveContainer" containerID="7eae71a6b3d368a666a8e96a2122aebb589721bbb3d60a95b0472b694b5d55c7" Jul 28 20:40:45.951664 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:45.951623 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:40:52.215293 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.215260 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg"] Jul 28 20:40:52.215861 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.215476 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" podUID="0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" containerName="payload-processing" containerID="cri-o://89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9" gracePeriod=30 Jul 28 20:40:52.225988 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.225960 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd"] Jul 28 20:40:52.226353 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.226311 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" podUID="d9a81bb1-a643-4e3f-bd55-0e85007f4a80" containerName="payload-pre-processing" containerID="cri-o://42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2" gracePeriod=30 Jul 28 20:40:52.479898 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.479836 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:40:52.511488 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.511462 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:40:52.562788 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.562756 2581 generic.go:356] "Generic (PLEG): container finished" podID="d9a81bb1-a643-4e3f-bd55-0e85007f4a80" containerID="42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2" exitCode=0 Jul 28 20:40:52.562927 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.562793 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" Jul 28 20:40:52.562927 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.562844 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" event={"ID":"d9a81bb1-a643-4e3f-bd55-0e85007f4a80","Type":"ContainerDied","Data":"42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2"} Jul 28 20:40:52.562927 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.562892 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd" event={"ID":"d9a81bb1-a643-4e3f-bd55-0e85007f4a80","Type":"ContainerDied","Data":"2fc5d4363fff35e11943123a6aeecf43cb629c8a83373f75463ac88b7f9a67cb"} Jul 28 20:40:52.562927 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.562911 2581 scope.go:117] "RemoveContainer" containerID="42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2" Jul 28 20:40:52.564228 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.564210 2581 generic.go:356] "Generic (PLEG): container finished" podID="0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" containerID="89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9" exitCode=0 Jul 28 20:40:52.564308 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.564242 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" event={"ID":"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4","Type":"ContainerDied","Data":"89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9"} Jul 28 20:40:52.564308 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.564291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" event={"ID":"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4","Type":"ContainerDied","Data":"eebf9b4b24437716fa034b7d6cb49b835474931aba7651c0509f26f958483b3a"} Jul 28 20:40:52.564308 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.564255 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg" Jul 28 20:40:52.571698 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.571683 2581 scope.go:117] "RemoveContainer" containerID="42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2" Jul 28 20:40:52.571952 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:40:52.571932 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2\": container with ID starting with 42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2 not found: ID does not exist" containerID="42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2" Jul 28 20:40:52.572004 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.571961 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2"} err="failed to get container status \"42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2\": rpc error: code = NotFound desc = could not find container \"42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2\": container with ID starting with 42bff0029e2bd4dc7f1a893641433ab35c107346721fbf73735ce1dfb67c5cd2 not found: ID does not exist" Jul 28 20:40:52.572004 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.571977 2581 scope.go:117] "RemoveContainer" containerID="89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9" Jul 28 20:40:52.579836 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.579818 2581 scope.go:117] "RemoveContainer" containerID="89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9" Jul 28 20:40:52.580082 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:40:52.580065 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9\": container with ID starting with 89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9 not found: ID does not exist" containerID="89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9" Jul 28 20:40:52.580141 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.580091 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9"} err="failed to get container status \"89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9\": rpc error: code = NotFound desc = could not find container \"89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9\": container with ID starting with 89f89a2d7afb505f76784d3ddb7182dc840421fdd31fe0b8928297b217d60fb9 not found: ID does not exist" Jul 28 20:40:52.596399 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.596380 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-plugins-config-volume\") pod \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " Jul 28 20:40:52.596528 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.596516 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tfwk\" (UniqueName: \"kubernetes.io/projected/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-kube-api-access-7tfwk\") pod \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\" (UID: \"0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4\") " Jul 28 20:40:52.596771 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.596747 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" (UID: "0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:40:52.598460 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.598441 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-kube-api-access-7tfwk" (OuterVolumeSpecName: "kube-api-access-7tfwk") pod "0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" (UID: "0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4"). InnerVolumeSpecName "kube-api-access-7tfwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:40:52.697935 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.697913 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-plugins-config-volume\") pod \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " Jul 28 20:40:52.698008 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.697942 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mckgw\" (UniqueName: \"kubernetes.io/projected/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-kube-api-access-mckgw\") pod \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\" (UID: \"d9a81bb1-a643-4e3f-bd55-0e85007f4a80\") " Jul 28 20:40:52.698221 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.698203 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tfwk\" (UniqueName: \"kubernetes.io/projected/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-kube-api-access-7tfwk\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:40:52.698266 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.698229 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:40:52.698298 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.698272 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9a81bb1-a643-4e3f-bd55-0e85007f4a80" (UID: "d9a81bb1-a643-4e3f-bd55-0e85007f4a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:40:52.699717 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.699698 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-kube-api-access-mckgw" (OuterVolumeSpecName: "kube-api-access-mckgw") pod "d9a81bb1-a643-4e3f-bd55-0e85007f4a80" (UID: "d9a81bb1-a643-4e3f-bd55-0e85007f4a80"). InnerVolumeSpecName "kube-api-access-mckgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:40:52.799164 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.799142 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:40:52.799164 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.799161 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mckgw\" (UniqueName: \"kubernetes.io/projected/d9a81bb1-a643-4e3f-bd55-0e85007f4a80-kube-api-access-mckgw\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:40:52.891053 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.891030 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg"] Jul 28 20:40:52.899279 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.899258 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ff6b3-5d98f4d674-p9fxg"] Jul 28 20:40:52.913544 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.913523 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd"] Jul 28 20:40:52.919617 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:52.919596 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ff6b3-5c7c949d84-c9rqd"] Jul 28 20:40:53.152243 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:53.152174 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" path="/var/lib/kubelet/pods/0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4/volumes" Jul 28 20:40:53.152557 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:40:53.152542 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a81bb1-a643-4e3f-bd55-0e85007f4a80" path="/var/lib/kubelet/pods/d9a81bb1-a643-4e3f-bd55-0e85007f4a80/volumes" Jul 28 20:41:01.396110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:01.396077 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:41:09.904135 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:09.904100 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx"] Jul 28 20:41:09.904614 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:09.904438 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" podUID="1ce01018-f26b-4ca7-bcbd-f08903df8d4e" containerName="payload-processing" containerID="cri-o://154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22" gracePeriod=30 Jul 28 20:41:09.913975 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:09.913949 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr"] Jul 28 20:41:09.914523 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:09.914495 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" podUID="90978197-8f8b-405d-9fb5-1270ebc1ecea" containerName="payload-pre-processing" containerID="cri-o://63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10" gracePeriod=30 Jul 28 20:41:10.173842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.173816 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:41:10.204046 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.204025 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:41:10.235001 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.234972 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-plugins-config-volume\") pod \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " Jul 28 20:41:10.235124 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.235108 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggq8g\" (UniqueName: \"kubernetes.io/projected/90978197-8f8b-405d-9fb5-1270ebc1ecea-kube-api-access-ggq8g\") pod \"90978197-8f8b-405d-9fb5-1270ebc1ecea\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " Jul 28 20:41:10.235181 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.235152 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90978197-8f8b-405d-9fb5-1270ebc1ecea-plugins-config-volume\") pod \"90978197-8f8b-405d-9fb5-1270ebc1ecea\" (UID: \"90978197-8f8b-405d-9fb5-1270ebc1ecea\") " Jul 28 20:41:10.235238 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.235180 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-245fl\" (UniqueName: \"kubernetes.io/projected/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-kube-api-access-245fl\") pod \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\" (UID: \"1ce01018-f26b-4ca7-bcbd-f08903df8d4e\") " Jul 28 20:41:10.235353 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.235301 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ce01018-f26b-4ca7-bcbd-f08903df8d4e" (UID: "1ce01018-f26b-4ca7-bcbd-f08903df8d4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:41:10.235911 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.235560 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90978197-8f8b-405d-9fb5-1270ebc1ecea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90978197-8f8b-405d-9fb5-1270ebc1ecea" (UID: "90978197-8f8b-405d-9fb5-1270ebc1ecea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:41:10.235911 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.235731 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:10.235911 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.235757 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90978197-8f8b-405d-9fb5-1270ebc1ecea-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:10.237267 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.237244 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-kube-api-access-245fl" (OuterVolumeSpecName: "kube-api-access-245fl") pod "1ce01018-f26b-4ca7-bcbd-f08903df8d4e" (UID: "1ce01018-f26b-4ca7-bcbd-f08903df8d4e"). InnerVolumeSpecName "kube-api-access-245fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:41:10.237423 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.237275 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90978197-8f8b-405d-9fb5-1270ebc1ecea-kube-api-access-ggq8g" (OuterVolumeSpecName: "kube-api-access-ggq8g") pod "90978197-8f8b-405d-9fb5-1270ebc1ecea" (UID: "90978197-8f8b-405d-9fb5-1270ebc1ecea"). InnerVolumeSpecName "kube-api-access-ggq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:41:10.336719 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.336680 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggq8g\" (UniqueName: \"kubernetes.io/projected/90978197-8f8b-405d-9fb5-1270ebc1ecea-kube-api-access-ggq8g\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:10.336719 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.336716 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245fl\" (UniqueName: \"kubernetes.io/projected/1ce01018-f26b-4ca7-bcbd-f08903df8d4e-kube-api-access-245fl\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:10.669241 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.669209 2581 generic.go:356] "Generic (PLEG): container finished" podID="1ce01018-f26b-4ca7-bcbd-f08903df8d4e" containerID="154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22" exitCode=0 Jul 28 20:41:10.669453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.669260 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" Jul 28 20:41:10.669453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.669303 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" event={"ID":"1ce01018-f26b-4ca7-bcbd-f08903df8d4e","Type":"ContainerDied","Data":"154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22"} Jul 28 20:41:10.669453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.669387 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx" event={"ID":"1ce01018-f26b-4ca7-bcbd-f08903df8d4e","Type":"ContainerDied","Data":"63c5e16647db529820722f28c346b23dbb8ba256d62d4eb0ae8c756f4b1e8ce9"} Jul 28 20:41:10.669453 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.669421 2581 scope.go:117] "RemoveContainer" containerID="154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22" Jul 28 20:41:10.670649 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.670625 2581 generic.go:356] "Generic (PLEG): container finished" podID="90978197-8f8b-405d-9fb5-1270ebc1ecea" containerID="63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10" exitCode=0 Jul 28 20:41:10.670742 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.670665 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" Jul 28 20:41:10.670742 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.670697 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" event={"ID":"90978197-8f8b-405d-9fb5-1270ebc1ecea","Type":"ContainerDied","Data":"63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10"} Jul 28 20:41:10.670742 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.670731 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr" event={"ID":"90978197-8f8b-405d-9fb5-1270ebc1ecea","Type":"ContainerDied","Data":"41a8e8a41eec5f714c4d30e16f1457f2575f822a28427b6a6f4088ae3b1fc93e"} Jul 28 20:41:10.679115 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.679093 2581 scope.go:117] "RemoveContainer" containerID="154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22" Jul 28 20:41:10.679390 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:41:10.679371 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22\": container with ID starting with 154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22 not found: ID does not exist" containerID="154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22" Jul 28 20:41:10.679471 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.679398 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22"} err="failed to get container status \"154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22\": rpc error: code = NotFound desc = could not find container \"154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22\": container with ID starting with 154e45ee1a22f1eef9b3eb66c8ded4da7e9c0d93c74a45bb9c03e83508a98e22 not found: ID does not exist" Jul 28 20:41:10.679471 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.679414 2581 scope.go:117] "RemoveContainer" containerID="63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10" Jul 28 20:41:10.687680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.687661 2581 scope.go:117] "RemoveContainer" containerID="63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10" Jul 28 20:41:10.687920 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:41:10.687902 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10\": container with ID starting with 63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10 not found: ID does not exist" containerID="63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10" Jul 28 20:41:10.687968 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.687927 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10"} err="failed to get container status \"63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10\": rpc error: code = NotFound desc = could not find container \"63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10\": container with ID starting with 63cf7e270105dc1ce8e42503183629613e7fdc73445e5bcc64e76d9b5e36ff10 not found: ID does not exist" Jul 28 20:41:10.696421 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.696402 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr"] Jul 28 20:41:10.702315 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.702293 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d38c0f-5fb76d55c6-7mhwr"] Jul 28 20:41:10.714689 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.714668 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx"] Jul 28 20:41:10.718813 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:10.718792 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d38c0f-649ddc956-jx4rx"] Jul 28 20:41:11.152450 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:11.152418 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce01018-f26b-4ca7-bcbd-f08903df8d4e" path="/var/lib/kubelet/pods/1ce01018-f26b-4ca7-bcbd-f08903df8d4e/volumes" Jul 28 20:41:11.152883 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:11.152817 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90978197-8f8b-405d-9fb5-1270ebc1ecea" path="/var/lib/kubelet/pods/90978197-8f8b-405d-9fb5-1270ebc1ecea/volumes" Jul 28 20:41:12.693401 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:12.693366 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:41:16.790697 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:16.790666 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p"] Jul 28 20:41:16.791090 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:16.790875 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" containerID="cri-o://c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30" gracePeriod=30 Jul 28 20:41:22.042651 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.042625 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:41:22.143369 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143282 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-socket\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143495 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143367 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtlx2\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-kube-api-access-rtlx2\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143495 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143389 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-credential-socket\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143495 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143409 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-data\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143495 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143448 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-envoy\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143495 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143481 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d8cc1c-5768-48de-9963-1d63a6516887-istiod-ca-cert\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143749 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143511 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-istio-token\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143749 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143515 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:41:22.143749 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143552 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-certs\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143749 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143582 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d8cc1c-5768-48de-9963-1d63a6516887-istio-podinfo\") pod \"33d8cc1c-5768-48de-9963-1d63a6516887\" (UID: \"33d8cc1c-5768-48de-9963-1d63a6516887\") " Jul 28 20:41:22.143749 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143709 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:41:22.144018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143864 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:41:22.144018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143941 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.144018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143961 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-credential-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.144018 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.143977 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-workload-certs\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.144224 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.144117 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d8cc1c-5768-48de-9963-1d63a6516887-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:41:22.144277 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.144232 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-data" (OuterVolumeSpecName: "istio-data") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:41:22.145634 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.145607 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:41:22.145750 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.145706 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33d8cc1c-5768-48de-9963-1d63a6516887-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:41:22.145935 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.145912 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-kube-api-access-rtlx2" (OuterVolumeSpecName: "kube-api-access-rtlx2") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "kube-api-access-rtlx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:41:22.145996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.145930 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-istio-token" (OuterVolumeSpecName: "istio-token") pod "33d8cc1c-5768-48de-9963-1d63a6516887" (UID: "33d8cc1c-5768-48de-9963-1d63a6516887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:41:22.244752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.244728 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtlx2\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-kube-api-access-rtlx2\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.244752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.244752 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-data\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.244933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.244762 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d8cc1c-5768-48de-9963-1d63a6516887-istio-envoy\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.244933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.244771 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d8cc1c-5768-48de-9963-1d63a6516887-istiod-ca-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.244933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.244779 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d8cc1c-5768-48de-9963-1d63a6516887-istio-token\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.244933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.244787 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d8cc1c-5768-48de-9963-1d63a6516887-istio-podinfo\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:41:22.376238 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376207 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7"] Jul 28 20:41:22.376656 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376635 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" containerName="payload-processing" Jul 28 20:41:22.376656 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376655 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" containerName="payload-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376669 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a81bb1-a643-4e3f-bd55-0e85007f4a80" containerName="payload-pre-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376676 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a81bb1-a643-4e3f-bd55-0e85007f4a80" containerName="payload-pre-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376684 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce01018-f26b-4ca7-bcbd-f08903df8d4e" containerName="payload-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376690 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce01018-f26b-4ca7-bcbd-f08903df8d4e" containerName="payload-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376700 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376705 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376712 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376718 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376730 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90978197-8f8b-405d-9fb5-1270ebc1ecea" containerName="payload-pre-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376735 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="90978197-8f8b-405d-9fb5-1270ebc1ecea" containerName="payload-pre-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376803 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="90978197-8f8b-405d-9fb5-1270ebc1ecea" containerName="payload-pre-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376813 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a81bb1-a643-4e3f-bd55-0e85007f4a80" containerName="payload-pre-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376819 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bfa55f4-53f2-4a11-81de-d3de1ebbf8a4" containerName="payload-processing" Jul 28 20:41:22.376819 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376826 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" Jul 28 20:41:22.377258 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.376832 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce01018-f26b-4ca7-bcbd-f08903df8d4e" containerName="payload-processing" Jul 28 20:41:22.380042 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.380026 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.382478 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.382457 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c55858\"" Jul 28 20:41:22.393078 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.393059 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz"] Jul 28 20:41:22.393492 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.393447 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" containerName="istio-proxy" Jul 28 20:41:22.396224 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.396209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.402408 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.402380 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c55858-dockercfg-cswv9\"" Jul 28 20:41:22.406966 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.406944 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7"] Jul 28 20:41:22.428068 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.428048 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz"] Jul 28 20:41:22.446416 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.446388 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae834d7-2d8f-429d-9df0-509c543e9ea1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c55858-66f6559d8-nclk7\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.446582 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.446505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2tj\" (UniqueName: \"kubernetes.io/projected/bae834d7-2d8f-429d-9df0-509c543e9ea1-kube-api-access-zd2tj\") pod \"payload-processing-e2e-api-a-c55858-66f6559d8-nclk7\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.547348 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.547299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgb9c\" (UniqueName: \"kubernetes.io/projected/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-kube-api-access-wgb9c\") pod \"payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.547535 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.547487 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae834d7-2d8f-429d-9df0-509c543e9ea1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c55858-66f6559d8-nclk7\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.547707 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.547678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2tj\" (UniqueName: \"kubernetes.io/projected/bae834d7-2d8f-429d-9df0-509c543e9ea1-kube-api-access-zd2tj\") pod \"payload-processing-e2e-api-a-c55858-66f6559d8-nclk7\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.547846 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.547747 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.548186 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.548166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae834d7-2d8f-429d-9df0-509c543e9ea1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c55858-66f6559d8-nclk7\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.560861 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.560832 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2tj\" (UniqueName: \"kubernetes.io/projected/bae834d7-2d8f-429d-9df0-509c543e9ea1-kube-api-access-zd2tj\") pod \"payload-processing-e2e-api-a-c55858-66f6559d8-nclk7\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.649197 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.649131 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.649197 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.649182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgb9c\" (UniqueName: \"kubernetes.io/projected/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-kube-api-access-wgb9c\") pod \"payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.649777 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.649759 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.666043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.666023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgb9c\" (UniqueName: \"kubernetes.io/projected/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-kube-api-access-wgb9c\") pod \"payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.690420 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.690402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:22.707116 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.706902 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:22.742931 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.742901 2581 generic.go:356] "Generic (PLEG): container finished" podID="33d8cc1c-5768-48de-9963-1d63a6516887" containerID="c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30" exitCode=0 Jul 28 20:41:22.743060 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.742969 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" Jul 28 20:41:22.743060 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.743003 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" event={"ID":"33d8cc1c-5768-48de-9963-1d63a6516887","Type":"ContainerDied","Data":"c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30"} Jul 28 20:41:22.743145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.743089 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p" event={"ID":"33d8cc1c-5768-48de-9963-1d63a6516887","Type":"ContainerDied","Data":"abf7b55730bbd7b3b094be83e45570a3d9748eb72003c3f2f4aaea06936f7682"} Jul 28 20:41:22.743145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.743129 2581 scope.go:117] "RemoveContainer" containerID="c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30" Jul 28 20:41:22.757156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.757092 2581 scope.go:117] "RemoveContainer" containerID="c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90" Jul 28 20:41:22.770024 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.769603 2581 scope.go:117] "RemoveContainer" containerID="c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30" Jul 28 20:41:22.770024 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:41:22.769984 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30\": container with ID starting with c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30 not found: ID does not exist" containerID="c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30" Jul 28 20:41:22.770223 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.770018 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30"} err="failed to get container status \"c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30\": rpc error: code = NotFound desc = could not find container \"c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30\": container with ID starting with c00853cf179f5b3a2679194c170a05c000a1ba5dbe7d9ecddb3ee0be7fc6ba30 not found: ID does not exist" Jul 28 20:41:22.770223 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.770046 2581 scope.go:117] "RemoveContainer" containerID="c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90" Jul 28 20:41:22.770438 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:41:22.770414 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90\": container with ID starting with c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90 not found: ID does not exist" containerID="c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90" Jul 28 20:41:22.770517 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.770461 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90"} err="failed to get container status \"c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90\": rpc error: code = NotFound desc = could not find container \"c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90\": container with ID starting with c0a26285e766698926f51b908bcd24d549ff037415ee7fb0487d83fb1c1f8a90 not found: ID does not exist" Jul 28 20:41:22.799379 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.799354 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p"] Jul 28 20:41:22.813625 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.813576 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d38c0f-openshift-default-7b476d7c84-4tz2p"] Jul 28 20:41:22.867195 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.867172 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7"] Jul 28 20:41:22.870062 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:41:22.870033 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae834d7_2d8f_429d_9df0_509c543e9ea1.slice/crio-ac17cb601e1eafff0b199b3f9568d8afbf61ca8ba8ed7d8a80c843034638c54d WatchSource:0}: Error finding container ac17cb601e1eafff0b199b3f9568d8afbf61ca8ba8ed7d8a80c843034638c54d: Status 404 returned error can't find the container with id ac17cb601e1eafff0b199b3f9568d8afbf61ca8ba8ed7d8a80c843034638c54d Jul 28 20:41:22.891095 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:22.891037 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz"] Jul 28 20:41:22.893609 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:41:22.893580 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e8f158_20c8_42cf_8ca7_24ab39c1f51a.slice/crio-fa9d2872af623d73a451a68783bf18ad79efdc499adfef7f0cb25e00c107ba40 WatchSource:0}: Error finding container fa9d2872af623d73a451a68783bf18ad79efdc499adfef7f0cb25e00c107ba40: Status 404 returned error can't find the container with id fa9d2872af623d73a451a68783bf18ad79efdc499adfef7f0cb25e00c107ba40 Jul 28 20:41:23.158209 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.158121 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d8cc1c-5768-48de-9963-1d63a6516887" path="/var/lib/kubelet/pods/33d8cc1c-5768-48de-9963-1d63a6516887/volumes" Jul 28 20:41:23.750172 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.750132 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" event={"ID":"bae834d7-2d8f-429d-9df0-509c543e9ea1","Type":"ContainerStarted","Data":"fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a"} Jul 28 20:41:23.750387 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.750201 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:23.750387 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.750223 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" event={"ID":"bae834d7-2d8f-429d-9df0-509c543e9ea1","Type":"ContainerStarted","Data":"ac17cb601e1eafff0b199b3f9568d8afbf61ca8ba8ed7d8a80c843034638c54d"} Jul 28 20:41:23.756161 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.756125 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" event={"ID":"53e8f158-20c8-42cf-8ca7-24ab39c1f51a","Type":"ContainerStarted","Data":"32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85"} Jul 28 20:41:23.756307 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.756181 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" event={"ID":"53e8f158-20c8-42cf-8ca7-24ab39c1f51a","Type":"ContainerStarted","Data":"fa9d2872af623d73a451a68783bf18ad79efdc499adfef7f0cb25e00c107ba40"} Jul 28 20:41:23.756307 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.756271 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:23.769894 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.769850 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" podStartSLOduration=1.76983662 podStartE2EDuration="1.76983662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:23.767343555 +0000 UTC m=+2511.211654563" watchObservedRunningTime="2026-07-28 20:41:23.76983662 +0000 UTC m=+2511.214147635" Jul 28 20:41:23.787192 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:23.787148 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" podStartSLOduration=1.787135326 podStartE2EDuration="1.787135326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:23.785508909 +0000 UTC m=+2511.229819925" watchObservedRunningTime="2026-07-28 20:41:23.787135326 +0000 UTC m=+2511.231446342" Jul 28 20:41:25.400632 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:25.400551 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:41:34.763315 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:34.763282 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:41:34.763718 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:34.763441 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:41:46.435439 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.435401 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9"] Jul 28 20:41:46.438912 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.438890 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.441747 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.441729 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2ed465-openshift-default-dockercfg-bmmdq\"" Jul 28 20:41:46.463153 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.463122 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9"] Jul 28 20:41:46.548486 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-token\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548665 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istiod-ca-cert\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548665 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548519 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmk6\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-kube-api-access-6gmk6\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548665 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548621 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-socket\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548665 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-credential-socket\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-certs\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-envoy\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-data\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.548850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.548732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-podinfo\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.649960 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.649932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmk6\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-kube-api-access-6gmk6\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650121 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.649975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-socket\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650121 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-credential-socket\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650121 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650049 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-certs\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650121 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650073 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-envoy\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650121 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650106 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-data\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650139 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-podinfo\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-token\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istiod-ca-cert\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650399 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-socket\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650614 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650471 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-certs\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650614 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650509 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-data\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650733 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650714 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-credential-socket\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.650913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.650891 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istiod-ca-cert\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.652513 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.652492 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-envoy\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.652752 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.652733 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-podinfo\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.665779 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.665748 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmk6\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-kube-api-access-6gmk6\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.668158 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.668136 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-token\") pod \"e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.749143 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.749112 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:46.906951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.906894 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9"] Jul 28 20:41:46.908488 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:41:46.908465 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f926a2_6af2_486d_838a_f7952a9dc5cf.slice/crio-2838732fa49118bb5272c394a0d5b7630ac46c95324ea38dd701d0edb39ca038 WatchSource:0}: Error finding container 2838732fa49118bb5272c394a0d5b7630ac46c95324ea38dd701d0edb39ca038: Status 404 returned error can't find the container with id 2838732fa49118bb5272c394a0d5b7630ac46c95324ea38dd701d0edb39ca038 Jul 28 20:41:46.910746 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.910717 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:41:46.910825 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.910776 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:41:46.910825 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:46.910804 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:41:47.888955 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:47.888918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" event={"ID":"c6f926a2-6af2-486d-838a-f7952a9dc5cf","Type":"ContainerStarted","Data":"2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f"} Jul 28 20:41:47.889369 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:47.888973 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" event={"ID":"c6f926a2-6af2-486d-838a-f7952a9dc5cf","Type":"ContainerStarted","Data":"2838732fa49118bb5272c394a0d5b7630ac46c95324ea38dd701d0edb39ca038"} Jul 28 20:41:47.913844 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:47.913788 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" podStartSLOduration=1.913772772 podStartE2EDuration="1.913772772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:47.911897855 +0000 UTC m=+2535.356208868" watchObservedRunningTime="2026-07-28 20:41:47.913772772 +0000 UTC m=+2535.358083841" Jul 28 20:41:48.749711 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:48.749671 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:48.754621 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:48.754593 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:48.894845 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:48.894815 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:48.895956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:48.895937 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:41:51.318591 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.318542 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm"] Jul 28 20:41:51.322481 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.322440 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.325417 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.325393 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ed465-dockercfg-pqsp5\"" Jul 28 20:41:51.325609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.325583 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ed465\"" Jul 28 20:41:51.345683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.345647 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm"] Jul 28 20:41:51.396638 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.396591 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.396638 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.396641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfks\" (UniqueName: \"kubernetes.io/projected/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-kube-api-access-jjfks\") pod \"payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.497777 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.497741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.497777 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.497783 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfks\" (UniqueName: \"kubernetes.io/projected/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-kube-api-access-jjfks\") pod \"payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.498478 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.498458 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.507264 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.507232 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfks\" (UniqueName: \"kubernetes.io/projected/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-kube-api-access-jjfks\") pod \"payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.633470 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.633378 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:51.797726 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.797686 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm"] Jul 28 20:41:51.798009 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:41:51.797986 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7d92d2_0779_44cd_9edb_5e9f2b0a8a1f.slice/crio-a5f872d9d84cac72d661da22e83bf99da138cc993e674627805565c9c24fd26e WatchSource:0}: Error finding container a5f872d9d84cac72d661da22e83bf99da138cc993e674627805565c9c24fd26e: Status 404 returned error can't find the container with id a5f872d9d84cac72d661da22e83bf99da138cc993e674627805565c9c24fd26e Jul 28 20:41:51.916278 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.916180 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" event={"ID":"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f","Type":"ContainerStarted","Data":"6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840"} Jul 28 20:41:51.916278 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.916238 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" event={"ID":"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f","Type":"ContainerStarted","Data":"a5f872d9d84cac72d661da22e83bf99da138cc993e674627805565c9c24fd26e"} Jul 28 20:41:51.916278 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:51.916260 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:41:52.030728 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:52.030671 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" podStartSLOduration=1.030655185 podStartE2EDuration="1.030655185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:41:51.975644655 +0000 UTC m=+2539.419955672" watchObservedRunningTime="2026-07-28 20:41:52.030655185 +0000 UTC m=+2539.474966200" Jul 28 20:41:55.026111 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:41:55.026070 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:42:02.924653 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:02.924626 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:42:02.979709 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:02.979674 2581 generic.go:356] "Generic (PLEG): container finished" podID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerID="2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f" exitCode=0 Jul 28 20:42:02.979881 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:02.979718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" event={"ID":"c6f926a2-6af2-486d-838a-f7952a9dc5cf","Type":"ContainerDied","Data":"2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f"} Jul 28 20:42:02.980122 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:02.980109 2581 scope.go:117] "RemoveContainer" containerID="2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f" Jul 28 20:42:02.981137 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:02.981109 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:42:02.981206 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:02.981192 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:42:02.981242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:02.981229 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:42:03.895221 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:03.895163 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:42:03.987668 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:03.987628 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" event={"ID":"c6f926a2-6af2-486d-838a-f7952a9dc5cf","Type":"ContainerStarted","Data":"2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392"} Jul 28 20:42:04.750380 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:04.750342 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:42:04.751621 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:04.751596 2581 patch_prober.go:28] interesting pod/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 20:42:04.751734 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:04.751657 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 20:42:05.750473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:05.750439 2581 patch_prober.go:28] interesting pod/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 20:42:05.750875 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:05.750496 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 20:42:06.749849 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:06.749808 2581 patch_prober.go:28] interesting pod/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 20:42:06.750029 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:06.749872 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 20:42:07.750384 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:07.750310 2581 patch_prober.go:28] interesting pod/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 20:42:07.750810 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:07.750407 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 20:42:08.753503 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:08.753468 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:42:08.753902 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:08.753751 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:42:08.754334 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:08.754301 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:42:33.638681 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:33.638640 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7"] Jul 28 20:42:33.639268 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:33.638873 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" podUID="bae834d7-2d8f-429d-9df0-509c543e9ea1" containerName="payload-processing" containerID="cri-o://fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a" gracePeriod=30 Jul 28 20:42:33.647454 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:33.647425 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz"] Jul 28 20:42:33.647647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:33.647611 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" podUID="53e8f158-20c8-42cf-8ca7-24ab39c1f51a" containerName="payload-pre-processing" containerID="cri-o://32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85" gracePeriod=30 Jul 28 20:42:33.929003 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:33.928977 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:42:33.932317 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:33.932294 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:42:34.084108 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.084071 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-plugins-config-volume\") pod \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " Jul 28 20:42:34.084309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.084158 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd2tj\" (UniqueName: \"kubernetes.io/projected/bae834d7-2d8f-429d-9df0-509c543e9ea1-kube-api-access-zd2tj\") pod \"bae834d7-2d8f-429d-9df0-509c543e9ea1\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " Jul 28 20:42:34.084309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.084207 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae834d7-2d8f-429d-9df0-509c543e9ea1-plugins-config-volume\") pod \"bae834d7-2d8f-429d-9df0-509c543e9ea1\" (UID: \"bae834d7-2d8f-429d-9df0-509c543e9ea1\") " Jul 28 20:42:34.084309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.084230 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgb9c\" (UniqueName: \"kubernetes.io/projected/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-kube-api-access-wgb9c\") pod \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\" (UID: \"53e8f158-20c8-42cf-8ca7-24ab39c1f51a\") " Jul 28 20:42:34.084603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.084571 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53e8f158-20c8-42cf-8ca7-24ab39c1f51a" (UID: "53e8f158-20c8-42cf-8ca7-24ab39c1f51a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:42:34.084603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.084590 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae834d7-2d8f-429d-9df0-509c543e9ea1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bae834d7-2d8f-429d-9df0-509c543e9ea1" (UID: "bae834d7-2d8f-429d-9df0-509c543e9ea1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:42:34.086368 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.086344 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-kube-api-access-wgb9c" (OuterVolumeSpecName: "kube-api-access-wgb9c") pod "53e8f158-20c8-42cf-8ca7-24ab39c1f51a" (UID: "53e8f158-20c8-42cf-8ca7-24ab39c1f51a"). InnerVolumeSpecName "kube-api-access-wgb9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:42:34.086446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.086381 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae834d7-2d8f-429d-9df0-509c543e9ea1-kube-api-access-zd2tj" (OuterVolumeSpecName: "kube-api-access-zd2tj") pod "bae834d7-2d8f-429d-9df0-509c543e9ea1" (UID: "bae834d7-2d8f-429d-9df0-509c543e9ea1"). InnerVolumeSpecName "kube-api-access-zd2tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:42:34.160205 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.160105 2581 generic.go:356] "Generic (PLEG): container finished" podID="bae834d7-2d8f-429d-9df0-509c543e9ea1" containerID="fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a" exitCode=0 Jul 28 20:42:34.160205 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.160154 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" Jul 28 20:42:34.160205 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.160192 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" event={"ID":"bae834d7-2d8f-429d-9df0-509c543e9ea1","Type":"ContainerDied","Data":"fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a"} Jul 28 20:42:34.160527 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.160248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7" event={"ID":"bae834d7-2d8f-429d-9df0-509c543e9ea1","Type":"ContainerDied","Data":"ac17cb601e1eafff0b199b3f9568d8afbf61ca8ba8ed7d8a80c843034638c54d"} Jul 28 20:42:34.160527 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.160277 2581 scope.go:117] "RemoveContainer" containerID="fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a" Jul 28 20:42:34.161699 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.161677 2581 generic.go:356] "Generic (PLEG): container finished" podID="53e8f158-20c8-42cf-8ca7-24ab39c1f51a" containerID="32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85" exitCode=0 Jul 28 20:42:34.161827 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.161724 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" Jul 28 20:42:34.161827 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.161731 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" event={"ID":"53e8f158-20c8-42cf-8ca7-24ab39c1f51a","Type":"ContainerDied","Data":"32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85"} Jul 28 20:42:34.161827 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.161765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz" event={"ID":"53e8f158-20c8-42cf-8ca7-24ab39c1f51a","Type":"ContainerDied","Data":"fa9d2872af623d73a451a68783bf18ad79efdc499adfef7f0cb25e00c107ba40"} Jul 28 20:42:34.179464 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.179442 2581 scope.go:117] "RemoveContainer" containerID="fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a" Jul 28 20:42:34.179788 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:42:34.179769 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a\": container with ID starting with fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a not found: ID does not exist" containerID="fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a" Jul 28 20:42:34.179842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.179797 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a"} err="failed to get container status \"fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a\": rpc error: code = NotFound desc = could not find container \"fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a\": container with ID starting with fc32439fc7d4586f0b7e20c82eee3ae6bce3391837fea5880399d94f1673551a not found: ID does not exist" Jul 28 20:42:34.179842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.179816 2581 scope.go:117] "RemoveContainer" containerID="32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85" Jul 28 20:42:34.185052 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.185028 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd2tj\" (UniqueName: \"kubernetes.io/projected/bae834d7-2d8f-429d-9df0-509c543e9ea1-kube-api-access-zd2tj\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:42:34.185160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.185054 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae834d7-2d8f-429d-9df0-509c543e9ea1-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:42:34.185160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.185068 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgb9c\" (UniqueName: \"kubernetes.io/projected/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-kube-api-access-wgb9c\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:42:34.185160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.185080 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e8f158-20c8-42cf-8ca7-24ab39c1f51a-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:42:34.188407 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.188388 2581 scope.go:117] "RemoveContainer" containerID="32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85" Jul 28 20:42:34.188710 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:42:34.188689 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85\": container with ID starting with 32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85 not found: ID does not exist" containerID="32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85" Jul 28 20:42:34.188760 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.188719 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85"} err="failed to get container status \"32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85\": rpc error: code = NotFound desc = could not find container \"32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85\": container with ID starting with 32e3f2efb7e20a69d946fabe4dc5351cf319812b4c4753844b70848965b25f85 not found: ID does not exist" Jul 28 20:42:34.201454 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.201420 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7"] Jul 28 20:42:34.208933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.208904 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c55858-66f6559d8-nclk7"] Jul 28 20:42:34.238628 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.238593 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz"] Jul 28 20:42:34.243345 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:34.243303 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c55858-fffc4b768-sh2fz"] Jul 28 20:42:35.153277 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:35.153243 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53e8f158-20c8-42cf-8ca7-24ab39c1f51a" path="/var/lib/kubelet/pods/53e8f158-20c8-42cf-8ca7-24ab39c1f51a/volumes" Jul 28 20:42:35.153690 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:35.153634 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bae834d7-2d8f-429d-9df0-509c543e9ea1" path="/var/lib/kubelet/pods/bae834d7-2d8f-429d-9df0-509c543e9ea1/volumes" Jul 28 20:42:35.999401 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:35.999356 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:42:42.796054 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.796010 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2"] Jul 28 20:42:42.796683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.796601 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae834d7-2d8f-429d-9df0-509c543e9ea1" containerName="payload-processing" Jul 28 20:42:42.796683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.796622 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae834d7-2d8f-429d-9df0-509c543e9ea1" containerName="payload-processing" Jul 28 20:42:42.796683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.796643 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e8f158-20c8-42cf-8ca7-24ab39c1f51a" containerName="payload-pre-processing" Jul 28 20:42:42.796683 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.796653 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e8f158-20c8-42cf-8ca7-24ab39c1f51a" containerName="payload-pre-processing" Jul 28 20:42:42.796914 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.796809 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bae834d7-2d8f-429d-9df0-509c543e9ea1" containerName="payload-processing" Jul 28 20:42:42.796914 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.796828 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="53e8f158-20c8-42cf-8ca7-24ab39c1f51a" containerName="payload-pre-processing" Jul 28 20:42:42.800010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.799988 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" Jul 28 20:42:42.802887 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.802864 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 20:42:42.803010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.802865 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:42:42.825500 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.825465 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2"] Jul 28 20:42:42.969804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:42.969759 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f309a62-21d3-4705-a58b-f80df912851f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2\" (UID: \"3f309a62-21d3-4705-a58b-f80df912851f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" Jul 28 20:42:43.070695 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:43.070595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f309a62-21d3-4705-a58b-f80df912851f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2\" (UID: \"3f309a62-21d3-4705-a58b-f80df912851f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" Jul 28 20:42:43.071243 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:43.071222 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f309a62-21d3-4705-a58b-f80df912851f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2\" (UID: \"3f309a62-21d3-4705-a58b-f80df912851f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" Jul 28 20:42:43.111423 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:43.111383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" Jul 28 20:42:43.284643 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:43.284617 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2"] Jul 28 20:42:43.286659 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:42:43.286622 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f309a62_21d3_4705_a58b_f80df912851f.slice/crio-6d645e24554462cd2e67a21ede75b053108888666a3d08c64dcc7bd50e01d506 WatchSource:0}: Error finding container 6d645e24554462cd2e67a21ede75b053108888666a3d08c64dcc7bd50e01d506: Status 404 returned error can't find the container with id 6d645e24554462cd2e67a21ede75b053108888666a3d08c64dcc7bd50e01d506 Jul 28 20:42:44.223638 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:44.223605 2581 generic.go:356] "Generic (PLEG): container finished" podID="3f309a62-21d3-4705-a58b-f80df912851f" containerID="3c3ab1314cf88bd7e4e304bb432d86f944f18a41ea2b19aa3e3d7437d3937ecf" exitCode=0 Jul 28 20:42:44.224012 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:44.223648 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" event={"ID":"3f309a62-21d3-4705-a58b-f80df912851f","Type":"ContainerDied","Data":"3c3ab1314cf88bd7e4e304bb432d86f944f18a41ea2b19aa3e3d7437d3937ecf"} Jul 28 20:42:44.224012 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:44.223710 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" event={"ID":"3f309a62-21d3-4705-a58b-f80df912851f","Type":"ContainerStarted","Data":"6d645e24554462cd2e67a21ede75b053108888666a3d08c64dcc7bd50e01d506"} Jul 28 20:42:45.367437 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:45.367404 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" Jul 28 20:42:45.497671 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:45.497631 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f309a62-21d3-4705-a58b-f80df912851f-maas-api-service-ca\") pod \"3f309a62-21d3-4705-a58b-f80df912851f\" (UID: \"3f309a62-21d3-4705-a58b-f80df912851f\") " Jul 28 20:42:45.498011 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:45.497985 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f309a62-21d3-4705-a58b-f80df912851f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f309a62-21d3-4705-a58b-f80df912851f" (UID: "3f309a62-21d3-4705-a58b-f80df912851f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:42:45.598612 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:45.598549 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f309a62-21d3-4705-a58b-f80df912851f-maas-api-service-ca\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:42:46.235865 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:46.235829 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" event={"ID":"3f309a62-21d3-4705-a58b-f80df912851f","Type":"ContainerDied","Data":"6d645e24554462cd2e67a21ede75b053108888666a3d08c64dcc7bd50e01d506"} Jul 28 20:42:46.235865 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:46.235865 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2" Jul 28 20:42:46.236077 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:46.235877 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d645e24554462cd2e67a21ede75b053108888666a3d08c64dcc7bd50e01d506" Jul 28 20:42:54.248506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.248471 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm"] Jul 28 20:42:54.248909 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.248707 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" podUID="2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" containerName="payload-processing" containerID="cri-o://6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840" gracePeriod=30 Jul 28 20:42:54.511221 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.511148 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:42:54.578994 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.578921 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjfks\" (UniqueName: \"kubernetes.io/projected/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-kube-api-access-jjfks\") pod \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " Jul 28 20:42:54.579120 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.579061 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-plugins-config-volume\") pod \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\" (UID: \"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f\") " Jul 28 20:42:54.579468 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.579442 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" (UID: "2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:42:54.581011 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.580976 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-kube-api-access-jjfks" (OuterVolumeSpecName: "kube-api-access-jjfks") pod "2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" (UID: "2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f"). InnerVolumeSpecName "kube-api-access-jjfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:42:54.657170 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.657133 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2"] Jul 28 20:42:54.659266 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.659241 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed465-8z9k2"] Jul 28 20:42:54.680327 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.680301 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:42:54.680456 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:54.680346 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjfks\" (UniqueName: \"kubernetes.io/projected/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f-kube-api-access-jjfks\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:42:55.152474 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.152442 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f309a62-21d3-4705-a58b-f80df912851f" path="/var/lib/kubelet/pods/3f309a62-21d3-4705-a58b-f80df912851f/volumes" Jul 28 20:42:55.287156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.287119 2581 generic.go:356] "Generic (PLEG): container finished" podID="2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" containerID="6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840" exitCode=0 Jul 28 20:42:55.287647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.287168 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" Jul 28 20:42:55.287647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.287204 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" event={"ID":"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f","Type":"ContainerDied","Data":"6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840"} Jul 28 20:42:55.287647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.287255 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm" event={"ID":"2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f","Type":"ContainerDied","Data":"a5f872d9d84cac72d661da22e83bf99da138cc993e674627805565c9c24fd26e"} Jul 28 20:42:55.287647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.287269 2581 scope.go:117] "RemoveContainer" containerID="6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840" Jul 28 20:42:55.295748 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.295733 2581 scope.go:117] "RemoveContainer" containerID="6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840" Jul 28 20:42:55.295980 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:42:55.295961 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840\": container with ID starting with 6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840 not found: ID does not exist" containerID="6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840" Jul 28 20:42:55.296035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.295987 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840"} err="failed to get container status \"6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840\": rpc error: code = NotFound desc = could not find container \"6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840\": container with ID starting with 6b4b745dfbce2108c4b780b92f8ecd72220f4d22021e6d5d722d20f17b7ff840 not found: ID does not exist" Jul 28 20:42:55.332398 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.332365 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm"] Jul 28 20:42:55.339973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:55.339947 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ed465-6556d4bc7d-chvbm"] Jul 28 20:42:56.212884 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:56.212837 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:42:57.152660 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:42:57.152626 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" path="/var/lib/kubelet/pods/2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f/volumes" Jul 28 20:43:01.846035 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:01.845986 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9"] Jul 28 20:43:01.846463 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:01.846224 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" containerID="cri-o://2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392" gracePeriod=30 Jul 28 20:43:07.099249 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.099221 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:43:07.199262 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199175 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-credential-socket\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199262 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199208 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-data\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199262 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199237 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gmk6\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-kube-api-access-6gmk6\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199546 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199271 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-socket\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199546 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199291 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-podinfo\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199546 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199377 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-certs\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199546 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199412 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-token\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199546 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199437 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istiod-ca-cert\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199802 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199545 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:43:07.199802 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199581 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-envoy\") pod \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\" (UID: \"c6f926a2-6af2-486d-838a-f7952a9dc5cf\") " Jul 28 20:43:07.199802 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199696 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:43:07.200139 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199749 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:43:07.200139 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199865 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:43:07.200334 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.199996 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-credential-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.200334 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.200267 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:43:07.201816 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.201784 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:43:07.201816 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.201801 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-kube-api-access-6gmk6" (OuterVolumeSpecName: "kube-api-access-6gmk6") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "kube-api-access-6gmk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:43:07.202003 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.201914 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:43:07.202003 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.201963 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6f926a2-6af2-486d-838a-f7952a9dc5cf" (UID: "c6f926a2-6af2-486d-838a-f7952a9dc5cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:43:07.300908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300868 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-data\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.300908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300902 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gmk6\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-kube-api-access-6gmk6\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.300908 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300917 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.301196 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300928 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-podinfo\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.301196 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300942 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-workload-certs\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.301196 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300953 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-token\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.301196 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300963 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istiod-ca-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.301196 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.300973 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f926a2-6af2-486d-838a-f7952a9dc5cf-istio-envoy\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:43:07.362724 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.362689 2581 generic.go:356] "Generic (PLEG): container finished" podID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerID="2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392" exitCode=0 Jul 28 20:43:07.362906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.362732 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" event={"ID":"c6f926a2-6af2-486d-838a-f7952a9dc5cf","Type":"ContainerDied","Data":"2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392"} Jul 28 20:43:07.362906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.362745 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" Jul 28 20:43:07.362906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.362795 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9" event={"ID":"c6f926a2-6af2-486d-838a-f7952a9dc5cf","Type":"ContainerDied","Data":"2838732fa49118bb5272c394a0d5b7630ac46c95324ea38dd701d0edb39ca038"} Jul 28 20:43:07.362906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.362816 2581 scope.go:117] "RemoveContainer" containerID="2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392" Jul 28 20:43:07.372644 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.372626 2581 scope.go:117] "RemoveContainer" containerID="2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f" Jul 28 20:43:07.381016 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.380991 2581 scope.go:117] "RemoveContainer" containerID="2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392" Jul 28 20:43:07.381291 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:43:07.381269 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392\": container with ID starting with 2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392 not found: ID does not exist" containerID="2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392" Jul 28 20:43:07.381368 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.381298 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392"} err="failed to get container status \"2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392\": rpc error: code = NotFound desc = could not find container \"2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392\": container with ID starting with 2edbb4ef9587e066d02d03ded890a3ba015aac06cfdee8deed0ef87feca00392 not found: ID does not exist" Jul 28 20:43:07.381368 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.381315 2581 scope.go:117] "RemoveContainer" containerID="2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f" Jul 28 20:43:07.381610 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:43:07.381591 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f\": container with ID starting with 2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f not found: ID does not exist" containerID="2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f" Jul 28 20:43:07.381666 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.381614 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f"} err="failed to get container status \"2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f\": rpc error: code = NotFound desc = could not find container \"2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f\": container with ID starting with 2b56f49bc4648d2a76467f3af2c6b53e505cb117bbad6aecde62d16865fdef6f not found: ID does not exist" Jul 28 20:43:07.425906 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.425869 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9"] Jul 28 20:43:07.426285 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.426263 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2ed465-openshift-default-8cd586cb6-zlxs9"] Jul 28 20:43:07.752967 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.752926 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg"] Jul 28 20:43:07.753390 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753368 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f309a62-21d3-4705-a58b-f80df912851f" containerName="revoke-keys" Jul 28 20:43:07.753506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753392 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f309a62-21d3-4705-a58b-f80df912851f" containerName="revoke-keys" Jul 28 20:43:07.753506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753414 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" Jul 28 20:43:07.753506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753424 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" Jul 28 20:43:07.753506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753445 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" Jul 28 20:43:07.753506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753453 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" Jul 28 20:43:07.753506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753471 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" containerName="payload-processing" Jul 28 20:43:07.753506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753481 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" containerName="payload-processing" Jul 28 20:43:07.753850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753585 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f309a62-21d3-4705-a58b-f80df912851f" containerName="revoke-keys" Jul 28 20:43:07.753850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753598 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" Jul 28 20:43:07.753850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.753611 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c7d92d2-0779-44cd-9edb-5e9f2b0a8a1f" containerName="payload-processing" Jul 28 20:43:07.758260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.758241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:07.761534 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.761510 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-1b6922-dockercfg-8r8xb\"" Jul 28 20:43:07.761744 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.761719 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-1b6922\"" Jul 28 20:43:07.771770 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.771746 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg"] Jul 28 20:43:07.806824 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.806786 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558c5b3c-542d-46ea-8b9d-bd3df13777bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:07.806985 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.806838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxm2t\" (UniqueName: \"kubernetes.io/projected/558c5b3c-542d-46ea-8b9d-bd3df13777bd-kube-api-access-kxm2t\") pod \"payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:07.821501 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.821468 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j"] Jul 28 20:43:07.821913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.821900 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" containerName="istio-proxy" Jul 28 20:43:07.824971 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.824957 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:07.850345 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.850283 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j"] Jul 28 20:43:07.907625 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.907585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxm2t\" (UniqueName: \"kubernetes.io/projected/558c5b3c-542d-46ea-8b9d-bd3df13777bd-kube-api-access-kxm2t\") pod \"payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:07.907842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.907657 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4186e0f0-0d24-44bc-bf46-dbae729e37ae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:07.907842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.907786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558c5b3c-542d-46ea-8b9d-bd3df13777bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:07.907842 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.907805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwtv\" (UniqueName: \"kubernetes.io/projected/4186e0f0-0d24-44bc-bf46-dbae729e37ae-kube-api-access-jzwtv\") pod \"payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:07.908569 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.908550 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558c5b3c-542d-46ea-8b9d-bd3df13777bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:07.916828 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:07.916802 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxm2t\" (UniqueName: \"kubernetes.io/projected/558c5b3c-542d-46ea-8b9d-bd3df13777bd-kube-api-access-kxm2t\") pod \"payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:08.009138 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.009036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwtv\" (UniqueName: \"kubernetes.io/projected/4186e0f0-0d24-44bc-bf46-dbae729e37ae-kube-api-access-jzwtv\") pod \"payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:08.009138 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.009105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4186e0f0-0d24-44bc-bf46-dbae729e37ae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:08.009844 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.009819 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4186e0f0-0d24-44bc-bf46-dbae729e37ae-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:08.019309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.019286 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwtv\" (UniqueName: \"kubernetes.io/projected/4186e0f0-0d24-44bc-bf46-dbae729e37ae-kube-api-access-jzwtv\") pod \"payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:08.069336 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.069293 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:08.135250 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.134989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:08.234355 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.234247 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg"] Jul 28 20:43:08.235590 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:43:08.235553 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558c5b3c_542d_46ea_8b9d_bd3df13777bd.slice/crio-4291b80ccb03a6961a366d5ec32fe4bdda315da9ba8bc7dc96296182e2289be8 WatchSource:0}: Error finding container 4291b80ccb03a6961a366d5ec32fe4bdda315da9ba8bc7dc96296182e2289be8: Status 404 returned error can't find the container with id 4291b80ccb03a6961a366d5ec32fe4bdda315da9ba8bc7dc96296182e2289be8 Jul 28 20:43:08.295565 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.295539 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j"] Jul 28 20:43:08.297395 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:43:08.297371 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4186e0f0_0d24_44bc_bf46_dbae729e37ae.slice/crio-39961f3e206f044e3932a23c23c8f6eedb87afad75b3f96c0e1db587e09eb8a7 WatchSource:0}: Error finding container 39961f3e206f044e3932a23c23c8f6eedb87afad75b3f96c0e1db587e09eb8a7: Status 404 returned error can't find the container with id 39961f3e206f044e3932a23c23c8f6eedb87afad75b3f96c0e1db587e09eb8a7 Jul 28 20:43:08.370392 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.370355 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" event={"ID":"558c5b3c-542d-46ea-8b9d-bd3df13777bd","Type":"ContainerStarted","Data":"f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f"} Jul 28 20:43:08.370566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.370423 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" event={"ID":"558c5b3c-542d-46ea-8b9d-bd3df13777bd","Type":"ContainerStarted","Data":"4291b80ccb03a6961a366d5ec32fe4bdda315da9ba8bc7dc96296182e2289be8"} Jul 28 20:43:08.370566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.370502 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:08.372186 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.372161 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" event={"ID":"4186e0f0-0d24-44bc-bf46-dbae729e37ae","Type":"ContainerStarted","Data":"67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356"} Jul 28 20:43:08.372407 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.372206 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" event={"ID":"4186e0f0-0d24-44bc-bf46-dbae729e37ae","Type":"ContainerStarted","Data":"39961f3e206f044e3932a23c23c8f6eedb87afad75b3f96c0e1db587e09eb8a7"} Jul 28 20:43:08.372407 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.372234 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:08.415372 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.415302 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" podStartSLOduration=1.415287382 podStartE2EDuration="1.415287382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:08.412034505 +0000 UTC m=+2615.856345520" watchObservedRunningTime="2026-07-28 20:43:08.415287382 +0000 UTC m=+2615.859598398" Jul 28 20:43:08.462222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:08.461208 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" podStartSLOduration=1.4611895879999999 podStartE2EDuration="1.461189588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:08.458898563 +0000 UTC m=+2615.903209580" watchObservedRunningTime="2026-07-28 20:43:08.461189588 +0000 UTC m=+2615.905500603" Jul 28 20:43:09.152929 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:09.152893 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f926a2-6af2-486d-838a-f7952a9dc5cf" path="/var/lib/kubelet/pods/c6f926a2-6af2-486d-838a-f7952a9dc5cf/volumes" Jul 28 20:43:11.311468 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:11.311430 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:43:19.381532 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:19.381494 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:43:19.382110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:19.381720 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:43:31.932268 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:31.932228 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s"] Jul 28 20:43:31.936127 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:31.936106 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:31.940532 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:31.940512 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c0e826-openshift-default-dockercfg-wbkqg\"" Jul 28 20:43:31.957135 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:31.957107 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s"] Jul 28 20:43:32.030186 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-envoy\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030416 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-podinfo\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030416 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-socket\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030416 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-token\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030416 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030348 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-credential-socket\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030601 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istiod-ca-cert\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030601 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj25f\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-kube-api-access-dj25f\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030601 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030512 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-data\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.030601 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.030581 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-certs\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131025 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.130988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-certs\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-envoy\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-podinfo\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-socket\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131434 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-token\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131434 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-credential-socket\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131434 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istiod-ca-cert\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj25f\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-kube-api-access-dj25f\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131515 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-certs\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131598 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-data\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131748 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-socket\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.131939 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131916 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-data\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.132012 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.131955 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-credential-socket\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.132081 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.132064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istiod-ca-cert\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.133518 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.133492 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-envoy\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.133723 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.133708 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-podinfo\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.144466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.144440 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-token\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.147919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.147891 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj25f\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-kube-api-access-dj25f\") pod \"e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.247022 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.246973 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:32.408400 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.408375 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s"] Jul 28 20:43:32.409644 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:43:32.409614 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51786bf7_54c3_43fd_8b40_b49fa5b2f280.slice/crio-e13851097dc43d098e3bcc4334ebaeb2616dce0e03a27f6af47b99c696575f51 WatchSource:0}: Error finding container e13851097dc43d098e3bcc4334ebaeb2616dce0e03a27f6af47b99c696575f51: Status 404 returned error can't find the container with id e13851097dc43d098e3bcc4334ebaeb2616dce0e03a27f6af47b99c696575f51 Jul 28 20:43:32.411755 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.411722 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:43:32.411851 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.411786 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:43:32.411851 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.411814 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:43:32.512895 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.512793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" event={"ID":"51786bf7-54c3-43fd-8b40-b49fa5b2f280","Type":"ContainerStarted","Data":"c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4"} Jul 28 20:43:32.512895 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.512861 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" event={"ID":"51786bf7-54c3-43fd-8b40-b49fa5b2f280","Type":"ContainerStarted","Data":"e13851097dc43d098e3bcc4334ebaeb2616dce0e03a27f6af47b99c696575f51"} Jul 28 20:43:32.551230 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:32.551177 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" podStartSLOduration=1.551159919 podStartE2EDuration="1.551159919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:32.550852428 +0000 UTC m=+2639.995163444" watchObservedRunningTime="2026-07-28 20:43:32.551159919 +0000 UTC m=+2639.995470935" Jul 28 20:43:33.247663 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:33.247627 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:33.253090 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:33.253062 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:33.518294 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:33.518211 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:33.519313 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:33.519294 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:36.029357 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.029308 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9"] Jul 28 20:43:36.034370 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.034350 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.037653 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.037633 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-c0e826-dockercfg-whw49\"" Jul 28 20:43:36.038256 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.038240 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-c0e826\"" Jul 28 20:43:36.046624 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.046574 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9"] Jul 28 20:43:36.062408 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.062373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkgp\" (UniqueName: \"kubernetes.io/projected/78296855-fbc4-4910-ac32-e9deda856149-kube-api-access-bbkgp\") pod \"payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.062541 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.062522 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78296855-fbc4-4910-ac32-e9deda856149-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.110650 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.110625 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq"] Jul 28 20:43:36.114163 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.114146 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.125967 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.125939 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq"] Jul 28 20:43:36.163302 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.163272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkgp\" (UniqueName: \"kubernetes.io/projected/78296855-fbc4-4910-ac32-e9deda856149-kube-api-access-bbkgp\") pod \"payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.163506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.163358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9tw\" (UniqueName: \"kubernetes.io/projected/9ef1f1de-b499-4926-af32-ffa76297d75e-kube-api-access-6n9tw\") pod \"payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.163506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.163408 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78296855-fbc4-4910-ac32-e9deda856149-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.163506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.163454 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef1f1de-b499-4926-af32-ffa76297d75e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.164083 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.164063 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78296855-fbc4-4910-ac32-e9deda856149-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.173248 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.173226 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkgp\" (UniqueName: \"kubernetes.io/projected/78296855-fbc4-4910-ac32-e9deda856149-kube-api-access-bbkgp\") pod \"payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.264446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.264389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9tw\" (UniqueName: \"kubernetes.io/projected/9ef1f1de-b499-4926-af32-ffa76297d75e-kube-api-access-6n9tw\") pod \"payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.264641 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.264520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef1f1de-b499-4926-af32-ffa76297d75e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.265185 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.265166 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef1f1de-b499-4926-af32-ffa76297d75e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.274962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.274942 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9tw\" (UniqueName: \"kubernetes.io/projected/9ef1f1de-b499-4926-af32-ffa76297d75e-kube-api-access-6n9tw\") pod \"payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.345019 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.344917 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:36.428837 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.428804 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:36.585942 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.585911 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq"] Jul 28 20:43:36.586081 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:43:36.586060 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef1f1de_b499_4926_af32_ffa76297d75e.slice/crio-2bec423d3fea4f62843cdca16d96378727605c581b5aa037df65513dba11942b WatchSource:0}: Error finding container 2bec423d3fea4f62843cdca16d96378727605c581b5aa037df65513dba11942b: Status 404 returned error can't find the container with id 2bec423d3fea4f62843cdca16d96378727605c581b5aa037df65513dba11942b Jul 28 20:43:36.720412 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:36.720304 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9"] Jul 28 20:43:36.722702 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:43:36.722675 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78296855_fbc4_4910_ac32_e9deda856149.slice/crio-c4f519dfe747aec9146493877ae79c16814e016a29c93d196ec2df052b6d40d0 WatchSource:0}: Error finding container c4f519dfe747aec9146493877ae79c16814e016a29c93d196ec2df052b6d40d0: Status 404 returned error can't find the container with id c4f519dfe747aec9146493877ae79c16814e016a29c93d196ec2df052b6d40d0 Jul 28 20:43:37.544437 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.544401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" event={"ID":"78296855-fbc4-4910-ac32-e9deda856149","Type":"ContainerStarted","Data":"1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66"} Jul 28 20:43:37.544872 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.544459 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" event={"ID":"78296855-fbc4-4910-ac32-e9deda856149","Type":"ContainerStarted","Data":"c4f519dfe747aec9146493877ae79c16814e016a29c93d196ec2df052b6d40d0"} Jul 28 20:43:37.544872 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.544597 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:37.545919 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.545896 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" event={"ID":"9ef1f1de-b499-4926-af32-ffa76297d75e","Type":"ContainerStarted","Data":"4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66"} Jul 28 20:43:37.546025 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.545941 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" event={"ID":"9ef1f1de-b499-4926-af32-ffa76297d75e","Type":"ContainerStarted","Data":"2bec423d3fea4f62843cdca16d96378727605c581b5aa037df65513dba11942b"} Jul 28 20:43:37.546025 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.545984 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:37.581637 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.581561 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" podStartSLOduration=1.581534924 podStartE2EDuration="1.581534924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:37.579181762 +0000 UTC m=+2645.023492780" watchObservedRunningTime="2026-07-28 20:43:37.581534924 +0000 UTC m=+2645.025845933" Jul 28 20:43:37.605131 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:37.605086 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" podStartSLOduration=1.605071464 podStartE2EDuration="1.605071464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:43:37.604241974 +0000 UTC m=+2645.048552994" watchObservedRunningTime="2026-07-28 20:43:37.605071464 +0000 UTC m=+2645.049382524" Jul 28 20:43:41.812216 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:41.812183 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:43:48.554536 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:48.554502 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:43:48.555049 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:48.554770 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:43:49.559508 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:49.559467 2581 patch_prober.go:28] interesting pod/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 20:43:49.559856 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:49.559568 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 20:43:50.621857 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:50.621820 2581 generic.go:356] "Generic (PLEG): container finished" podID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerID="c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4" exitCode=0 Jul 28 20:43:50.622467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:50.621896 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" event={"ID":"51786bf7-54c3-43fd-8b40-b49fa5b2f280","Type":"ContainerDied","Data":"c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4"} Jul 28 20:43:50.622567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:50.622545 2581 scope.go:117] "RemoveContainer" containerID="c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4" Jul 28 20:43:50.623660 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:50.623633 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:43:50.623759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:50.623696 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:43:50.623759 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:50.623727 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:43:51.633420 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:51.633378 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" event={"ID":"51786bf7-54c3-43fd-8b40-b49fa5b2f280","Type":"ContainerStarted","Data":"33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87"} Jul 28 20:43:52.248045 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:52.248010 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:52.248244 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:52.248223 2581 patch_prober.go:28] interesting pod/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 28 20:43:52.248291 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:52.248272 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 28 20:43:53.248052 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:53.248014 2581 patch_prober.go:28] interesting pod/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 28 20:43:53.248463 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:53.248072 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 28 20:43:54.247880 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:54.247844 2581 patch_prober.go:28] interesting pod/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 28 20:43:54.248062 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:54.247901 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 28 20:43:55.247869 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:55.247832 2581 patch_prober.go:28] interesting pod/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Jul 28 20:43:55.248264 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:55.247886 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Jul 28 20:43:56.251280 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:56.251246 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:56.251676 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:56.251517 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:43:56.252186 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:43:56.252170 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:44:08.286620 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:44:08.286584 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:44:52.392570 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:44:52.392530 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:45:00.151314 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:00.151279 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2"] Jul 28 20:45:00.154925 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:00.154906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" Jul 28 20:45:00.157118 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:00.157094 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 20:45:00.164767 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:00.164751 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" Jul 28 20:45:00.183503 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:00.183473 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2"] Jul 28 20:45:00.522651 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:00.522629 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2"] Jul 28 20:45:00.525181 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:45:00.525153 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59dbe31_75da_44a6_9de6_7360a4ba6e6d.slice/crio-012b6842fe4633231b1b75a22240e5a24c3bd94c5fc4a67e6adb0be520efe597 WatchSource:0}: Error finding container 012b6842fe4633231b1b75a22240e5a24c3bd94c5fc4a67e6adb0be520efe597: Status 404 returned error can't find the container with id 012b6842fe4633231b1b75a22240e5a24c3bd94c5fc4a67e6adb0be520efe597 Jul 28 20:45:01.033284 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:01.033253 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" event={"ID":"d59dbe31-75da-44a6-9de6-7360a4ba6e6d","Type":"ContainerStarted","Data":"b32d352fee7138616e6af1ea0b54cc0950e59c687087da3b2338871aefcdab0d"} Jul 28 20:45:01.033476 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:01.033302 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" event={"ID":"d59dbe31-75da-44a6-9de6-7360a4ba6e6d","Type":"ContainerStarted","Data":"012b6842fe4633231b1b75a22240e5a24c3bd94c5fc4a67e6adb0be520efe597"} Jul 28 20:45:01.062782 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:01.062687 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" podStartSLOduration=1.062672498 podStartE2EDuration="1.062672498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:45:01.060535402 +0000 UTC m=+2728.504846419" watchObservedRunningTime="2026-07-28 20:45:01.062672498 +0000 UTC m=+2728.506983567" Jul 28 20:45:02.040210 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:02.040174 2581 generic.go:356] "Generic (PLEG): container finished" podID="d59dbe31-75da-44a6-9de6-7360a4ba6e6d" containerID="b32d352fee7138616e6af1ea0b54cc0950e59c687087da3b2338871aefcdab0d" exitCode=0 Jul 28 20:45:02.040634 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:02.040261 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" event={"ID":"d59dbe31-75da-44a6-9de6-7360a4ba6e6d","Type":"ContainerDied","Data":"b32d352fee7138616e6af1ea0b54cc0950e59c687087da3b2338871aefcdab0d"} Jul 28 20:45:03.184467 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:03.184436 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" Jul 28 20:45:04.053853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:04.053820 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" event={"ID":"d59dbe31-75da-44a6-9de6-7360a4ba6e6d","Type":"ContainerDied","Data":"012b6842fe4633231b1b75a22240e5a24c3bd94c5fc4a67e6adb0be520efe597"} Jul 28 20:45:04.053853 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:04.053849 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2" Jul 28 20:45:04.054067 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:04.053871 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012b6842fe4633231b1b75a22240e5a24c3bd94c5fc4a67e6adb0be520efe597" Jul 28 20:45:16.522827 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:16.522783 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:45:44.207989 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:45:44.207953 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:46:01.016527 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:46:01.016449 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:46:27.699148 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:46:27.699110 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:46:43.998505 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:46:43.998465 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:47:06.736421 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:47:06.736375 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:47:24.697786 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:47:24.697692 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:47:51.594868 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:47:51.594832 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:48:10.791793 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:48:10.791751 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:48:33.113149 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:48:33.113114 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:48:51.493271 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:48:51.493229 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:49:16.890605 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:49:16.890518 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:49:28.797195 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:49:28.797161 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:49:33.497547 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:49:33.497516 2581 scope.go:117] "RemoveContainer" containerID="3c3ab1314cf88bd7e4e304bb432d86f944f18a41ea2b19aa3e3d7437d3937ecf" Jul 28 20:49:42.186921 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:49:42.186888 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:50:00.396407 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:50:00.396371 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:50:39.488870 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:50:39.488834 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:51:01.292891 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:01.292855 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:51:21.285553 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:21.285521 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:51:40.782631 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:40.782599 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:51:44.253615 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.253578 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9nn4v"] Jul 28 20:51:44.253991 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.253977 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59dbe31-75da-44a6-9de6-7360a4ba6e6d" containerName="cleanup" Jul 28 20:51:44.253991 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.253991 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59dbe31-75da-44a6-9de6-7360a4ba6e6d" containerName="cleanup" Jul 28 20:51:44.254110 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.254059 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59dbe31-75da-44a6-9de6-7360a4ba6e6d" containerName="cleanup" Jul 28 20:51:44.257103 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.257084 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:44.259591 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.259566 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-wxm2p\"" Jul 28 20:51:44.270222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.270196 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9nn4v"] Jul 28 20:51:44.383343 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.383284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbsq\" (UniqueName: \"kubernetes.io/projected/79e1543a-0d06-4d9b-a85e-67117f4e89a9-kube-api-access-6cbsq\") pod \"kuadrant-operator-catalog-9nn4v\" (UID: \"79e1543a-0d06-4d9b-a85e-67117f4e89a9\") " pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:44.484662 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.484628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbsq\" (UniqueName: \"kubernetes.io/projected/79e1543a-0d06-4d9b-a85e-67117f4e89a9-kube-api-access-6cbsq\") pod \"kuadrant-operator-catalog-9nn4v\" (UID: \"79e1543a-0d06-4d9b-a85e-67117f4e89a9\") " pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:44.494596 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.494567 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbsq\" (UniqueName: \"kubernetes.io/projected/79e1543a-0d06-4d9b-a85e-67117f4e89a9-kube-api-access-6cbsq\") pod \"kuadrant-operator-catalog-9nn4v\" (UID: \"79e1543a-0d06-4d9b-a85e-67117f4e89a9\") " pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:44.566750 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.566671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:44.730682 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.730648 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9nn4v"] Jul 28 20:51:44.731838 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:51:44.731813 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e1543a_0d06_4d9b_a85e_67117f4e89a9.slice/crio-f6c7c5e1d1175abe45afe5051a7a6f21fe2e23c78b8715f68bd54bb329f46ab6 WatchSource:0}: Error finding container f6c7c5e1d1175abe45afe5051a7a6f21fe2e23c78b8715f68bd54bb329f46ab6: Status 404 returned error can't find the container with id f6c7c5e1d1175abe45afe5051a7a6f21fe2e23c78b8715f68bd54bb329f46ab6 Jul 28 20:51:44.733145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:44.733123 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 20:51:45.371780 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:45.371732 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" event={"ID":"79e1543a-0d06-4d9b-a85e-67117f4e89a9","Type":"ContainerStarted","Data":"f6c7c5e1d1175abe45afe5051a7a6f21fe2e23c78b8715f68bd54bb329f46ab6"} Jul 28 20:51:47.385903 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:47.385868 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" event={"ID":"79e1543a-0d06-4d9b-a85e-67117f4e89a9","Type":"ContainerStarted","Data":"7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3"} Jul 28 20:51:47.414487 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:47.414436 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" podStartSLOduration=1.276280064 podStartE2EDuration="3.414421021s" totalImagesPullingTime="2.138140957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:51:47.411794627 +0000 UTC m=+3134.856105643" watchObservedRunningTime="2026-07-28 20:51:47.414421021 +0000 UTC m=+3134.858732036" Jul 28 20:51:54.567852 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:54.567760 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:54.567852 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:54.567817 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:54.589636 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:54.589599 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:55.451490 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:55.451461 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:55.510625 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:55.510592 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9nn4v"] Jul 28 20:51:57.441610 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:57.441552 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" podUID="79e1543a-0d06-4d9b-a85e-67117f4e89a9" containerName="registry-server" containerID="cri-o://7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3" gracePeriod=2 Jul 28 20:51:57.692928 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:57.692864 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:57.808147 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:57.808109 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cbsq\" (UniqueName: \"kubernetes.io/projected/79e1543a-0d06-4d9b-a85e-67117f4e89a9-kube-api-access-6cbsq\") pod \"79e1543a-0d06-4d9b-a85e-67117f4e89a9\" (UID: \"79e1543a-0d06-4d9b-a85e-67117f4e89a9\") " Jul 28 20:51:57.810168 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:57.810136 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e1543a-0d06-4d9b-a85e-67117f4e89a9-kube-api-access-6cbsq" (OuterVolumeSpecName: "kube-api-access-6cbsq") pod "79e1543a-0d06-4d9b-a85e-67117f4e89a9" (UID: "79e1543a-0d06-4d9b-a85e-67117f4e89a9"). InnerVolumeSpecName "kube-api-access-6cbsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:51:57.908836 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:57.908800 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbsq\" (UniqueName: \"kubernetes.io/projected/79e1543a-0d06-4d9b-a85e-67117f4e89a9-kube-api-access-6cbsq\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:51:58.448366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.448309 2581 generic.go:356] "Generic (PLEG): container finished" podID="79e1543a-0d06-4d9b-a85e-67117f4e89a9" containerID="7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3" exitCode=0 Jul 28 20:51:58.448366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.448356 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" event={"ID":"79e1543a-0d06-4d9b-a85e-67117f4e89a9","Type":"ContainerDied","Data":"7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3"} Jul 28 20:51:58.448915 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.448372 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" Jul 28 20:51:58.448915 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.448410 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9nn4v" event={"ID":"79e1543a-0d06-4d9b-a85e-67117f4e89a9","Type":"ContainerDied","Data":"f6c7c5e1d1175abe45afe5051a7a6f21fe2e23c78b8715f68bd54bb329f46ab6"} Jul 28 20:51:58.448915 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.448425 2581 scope.go:117] "RemoveContainer" containerID="7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3" Jul 28 20:51:58.458426 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.458409 2581 scope.go:117] "RemoveContainer" containerID="7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3" Jul 28 20:51:58.458709 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:51:58.458692 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3\": container with ID starting with 7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3 not found: ID does not exist" containerID="7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3" Jul 28 20:51:58.458756 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.458716 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3"} err="failed to get container status \"7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3\": rpc error: code = NotFound desc = could not find container \"7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3\": container with ID starting with 7c46cc25db1d5a01c420c159738cf243b39a384d5d9f015b84683d0abd80fde3 not found: ID does not exist" Jul 28 20:51:58.472595 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.472574 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9nn4v"] Jul 28 20:51:58.478087 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:58.478061 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9nn4v"] Jul 28 20:51:59.152987 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:51:59.152957 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e1543a-0d06-4d9b-a85e-67117f4e89a9" path="/var/lib/kubelet/pods/79e1543a-0d06-4d9b-a85e-67117f4e89a9/volumes" Jul 28 20:52:08.282371 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:52:08.282335 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:52:37.806699 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:52:37.806666 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:52:58.780988 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:52:58.780951 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:53:13.693606 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:13.693574 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:53:30.215410 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.215380 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88"] Jul 28 20:53:30.215796 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.215784 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e1543a-0d06-4d9b-a85e-67117f4e89a9" containerName="registry-server" Jul 28 20:53:30.215839 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.215797 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e1543a-0d06-4d9b-a85e-67117f4e89a9" containerName="registry-server" Jul 28 20:53:30.215883 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.215874 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e1543a-0d06-4d9b-a85e-67117f4e89a9" containerName="registry-server" Jul 28 20:53:30.218958 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.218911 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.220804 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.220786 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2dfc5b\"" Jul 28 20:53:30.229013 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.228985 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88"] Jul 28 20:53:30.245353 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.245308 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6"] Jul 28 20:53:30.249397 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.249380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.251994 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.251977 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2dfc5b-dockercfg-qx9rk\"" Jul 28 20:53:30.257026 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.257004 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6"] Jul 28 20:53:30.321635 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.321604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.321786 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.321670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcn2j\" (UniqueName: \"kubernetes.io/projected/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-kube-api-access-xcn2j\") pod \"payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.422745 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.422713 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.422745 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.422751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x546l\" (UniqueName: \"kubernetes.io/projected/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-kube-api-access-x546l\") pod \"payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.422951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.422822 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcn2j\" (UniqueName: \"kubernetes.io/projected/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-kube-api-access-xcn2j\") pod \"payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.422951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.422862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.423466 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.423447 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.431178 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.431151 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcn2j\" (UniqueName: \"kubernetes.io/projected/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-kube-api-access-xcn2j\") pod \"payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.523733 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.523696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x546l\" (UniqueName: \"kubernetes.io/projected/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-kube-api-access-x546l\") pod \"payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.523907 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.523791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.524507 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.524485 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.529780 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.529759 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:30.535608 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.535587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x546l\" (UniqueName: \"kubernetes.io/projected/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-kube-api-access-x546l\") pod \"payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.560585 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.560557 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.698525 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.698498 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88"] Jul 28 20:53:30.700270 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:53:30.700199 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939eece6_bffd_4bc9_9ea1_e4ff8e9f4c36.slice/crio-8d08438b37ca5c4622aa6fcc5489102a472a191de84cae56b8acb5b224011a69 WatchSource:0}: Error finding container 8d08438b37ca5c4622aa6fcc5489102a472a191de84cae56b8acb5b224011a69: Status 404 returned error can't find the container with id 8d08438b37ca5c4622aa6fcc5489102a472a191de84cae56b8acb5b224011a69 Jul 28 20:53:30.719440 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.719416 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6"] Jul 28 20:53:30.720948 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:53:30.720920 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19fdb3a_f626_4d8c_b7d9_daeefa9c0387.slice/crio-f19fc99585656b1cfd02f8635d2225ef142d3482ad4bad2396793be0b7737726 WatchSource:0}: Error finding container f19fc99585656b1cfd02f8635d2225ef142d3482ad4bad2396793be0b7737726: Status 404 returned error can't find the container with id f19fc99585656b1cfd02f8635d2225ef142d3482ad4bad2396793be0b7737726 Jul 28 20:53:30.989364 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.989306 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" event={"ID":"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387","Type":"ContainerStarted","Data":"c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8"} Jul 28 20:53:30.989556 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.989424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" event={"ID":"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387","Type":"ContainerStarted","Data":"f19fc99585656b1cfd02f8635d2225ef142d3482ad4bad2396793be0b7737726"} Jul 28 20:53:30.989556 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.989455 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:30.990816 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.990790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" event={"ID":"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36","Type":"ContainerStarted","Data":"a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6"} Jul 28 20:53:30.990938 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.990832 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" event={"ID":"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36","Type":"ContainerStarted","Data":"8d08438b37ca5c4622aa6fcc5489102a472a191de84cae56b8acb5b224011a69"} Jul 28 20:53:30.990938 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:30.990852 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:31.008470 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:31.008424 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" podStartSLOduration=1.008408872 podStartE2EDuration="1.008408872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:31.006201545 +0000 UTC m=+3238.450512564" watchObservedRunningTime="2026-07-28 20:53:31.008408872 +0000 UTC m=+3238.452719890" Jul 28 20:53:31.023709 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:31.023665 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" podStartSLOduration=1.023655522 podStartE2EDuration="1.023655522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:31.021498924 +0000 UTC m=+3238.465809955" watchObservedRunningTime="2026-07-28 20:53:31.023655522 +0000 UTC m=+3238.467966535" Jul 28 20:53:38.198526 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:38.198492 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:53:41.999145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:41.999115 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:53:41.999548 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:41.999284 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:53:50.984445 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:50.984417 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c"] Jul 28 20:53:50.988043 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:50.988024 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:50.991197 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:50.991173 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1ac855\"" Jul 28 20:53:50.991329 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:50.991224 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1ac855-dockercfg-bddp6\"" Jul 28 20:53:51.000956 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.000933 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c"] Jul 28 20:53:51.079619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.079585 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc"] Jul 28 20:53:51.083255 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.083239 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.092089 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.092063 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc"] Jul 28 20:53:51.116690 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.116662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5c2g\" (UniqueName: \"kubernetes.io/projected/427171fc-9f77-46d6-a210-63f7b27504d3-kube-api-access-z5c2g\") pod \"payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:51.116847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.116713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbftk\" (UniqueName: \"kubernetes.io/projected/7925da09-1920-463d-ab3c-4c48782cec9d-kube-api-access-wbftk\") pod \"payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.116847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.116825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7925da09-1920-463d-ab3c-4c48782cec9d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.116974 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.116917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427171fc-9f77-46d6-a210-63f7b27504d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:51.218015 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.217984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427171fc-9f77-46d6-a210-63f7b27504d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:51.218156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.218031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5c2g\" (UniqueName: \"kubernetes.io/projected/427171fc-9f77-46d6-a210-63f7b27504d3-kube-api-access-z5c2g\") pod \"payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:51.218156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.218076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbftk\" (UniqueName: \"kubernetes.io/projected/7925da09-1920-463d-ab3c-4c48782cec9d-kube-api-access-wbftk\") pod \"payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.218156 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.218114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7925da09-1920-463d-ab3c-4c48782cec9d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.218760 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.218741 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427171fc-9f77-46d6-a210-63f7b27504d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:51.218897 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.218880 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7925da09-1920-463d-ab3c-4c48782cec9d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.225682 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.225662 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbftk\" (UniqueName: \"kubernetes.io/projected/7925da09-1920-463d-ab3c-4c48782cec9d-kube-api-access-wbftk\") pod \"payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.225807 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.225788 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5c2g\" (UniqueName: \"kubernetes.io/projected/427171fc-9f77-46d6-a210-63f7b27504d3-kube-api-access-z5c2g\") pod \"payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:51.299131 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.299053 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:51.394933 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.394903 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:51.433242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.433212 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c"] Jul 28 20:53:51.434460 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:53:51.434434 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427171fc_9f77_46d6_a210_63f7b27504d3.slice/crio-040460b58ddac04d33d6e21f1b263ded5fc609f15baf777feb3d55be1c6f37eb WatchSource:0}: Error finding container 040460b58ddac04d33d6e21f1b263ded5fc609f15baf777feb3d55be1c6f37eb: Status 404 returned error can't find the container with id 040460b58ddac04d33d6e21f1b263ded5fc609f15baf777feb3d55be1c6f37eb Jul 28 20:53:51.751729 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:51.751701 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc"] Jul 28 20:53:51.753285 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:53:51.753253 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7925da09_1920_463d_ab3c_4c48782cec9d.slice/crio-ba81b94c29645a00a6323fe59ae31f489517a3a4a2221c83239230410baffdad WatchSource:0}: Error finding container ba81b94c29645a00a6323fe59ae31f489517a3a4a2221c83239230410baffdad: Status 404 returned error can't find the container with id ba81b94c29645a00a6323fe59ae31f489517a3a4a2221c83239230410baffdad Jul 28 20:53:52.123041 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.122945 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" event={"ID":"427171fc-9f77-46d6-a210-63f7b27504d3","Type":"ContainerStarted","Data":"d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8"} Jul 28 20:53:52.123041 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.123016 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" event={"ID":"427171fc-9f77-46d6-a210-63f7b27504d3","Type":"ContainerStarted","Data":"040460b58ddac04d33d6e21f1b263ded5fc609f15baf777feb3d55be1c6f37eb"} Jul 28 20:53:52.123604 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.123157 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:53:52.124578 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.124556 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" event={"ID":"7925da09-1920-463d-ab3c-4c48782cec9d","Type":"ContainerStarted","Data":"ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967"} Jul 28 20:53:52.124670 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.124591 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" event={"ID":"7925da09-1920-463d-ab3c-4c48782cec9d","Type":"ContainerStarted","Data":"ba81b94c29645a00a6323fe59ae31f489517a3a4a2221c83239230410baffdad"} Jul 28 20:53:52.124710 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.124678 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:53:52.149082 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.149021 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" podStartSLOduration=2.149003465 podStartE2EDuration="2.149003465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:52.13799843 +0000 UTC m=+3259.582309449" watchObservedRunningTime="2026-07-28 20:53:52.149003465 +0000 UTC m=+3259.593314460" Jul 28 20:53:52.160201 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:52.160162 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" podStartSLOduration=1.160149322 podStartE2EDuration="1.160149322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:53:52.15882938 +0000 UTC m=+3259.603140397" watchObservedRunningTime="2026-07-28 20:53:52.160149322 +0000 UTC m=+3259.604460337" Jul 28 20:53:57.195173 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:53:57.195136 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:54:03.132496 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:54:03.132463 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:54:03.132948 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:54:03.132531 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:54:10.135590 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:54:10.135558 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc_7925da09-1920-463d-ab3c-4c48782cec9d/payload-processing/0.log" Jul 28 20:54:31.686176 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:54:31.686141 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:55:00.385195 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:00.385120 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:55:12.429101 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:12.429072 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6_b19fdb3a-f626-4d8c-b7d9-daeefa9c0387/payload-processing/0.log" Jul 28 20:55:12.666913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:12.666877 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc_7925da09-1920-463d-ab3c-4c48782cec9d/payload-processing/0.log" Jul 28 20:55:13.440394 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.440334 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r"] Jul 28 20:55:13.444207 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.444186 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.452101 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.452082 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-d6a3fd-openshift-default-dockercfg-ffntv\"" Jul 28 20:55:13.460840 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.460818 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r"] Jul 28 20:55:13.502095 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9cf8\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-kube-api-access-s9cf8\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-envoy\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-certs\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502171 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-podinfo\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502242 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-token\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-data\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502269 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-credential-socket\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.502473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.502286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-socket\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.603537 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9cf8\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-kube-api-access-s9cf8\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.603715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-envoy\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.603715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-certs\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.603715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-podinfo\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.603715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.603715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-token\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.603715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-data\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.604003 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-credential-socket\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.604003 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.603749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-socket\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.604115 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.604051 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-certs\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.604212 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.604190 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-credential-socket\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.604278 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.604231 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-data\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.604317 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.604286 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-socket\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.604424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.604407 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istiod-ca-cert\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.605925 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.605907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-envoy\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.606381 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.606364 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-podinfo\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.614269 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.614241 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9cf8\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-kube-api-access-s9cf8\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.615305 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.615284 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-token\") pod \"e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.754502 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.754466 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:13.907183 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.907153 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r"] Jul 28 20:55:13.909030 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:55:13.909006 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab774a9_c389_4b5e_ae95_3ed75b46f741.slice/crio-ca8cdec49dad6572e5be41a7987845f1c0e8db4da9bc1ba8cf511aa10d1be987 WatchSource:0}: Error finding container ca8cdec49dad6572e5be41a7987845f1c0e8db4da9bc1ba8cf511aa10d1be987: Status 404 returned error can't find the container with id ca8cdec49dad6572e5be41a7987845f1c0e8db4da9bc1ba8cf511aa10d1be987 Jul 28 20:55:13.911192 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.911162 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:55:13.911259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.911226 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:55:13.911259 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:13.911255 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:55:14.612633 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:14.612597 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" event={"ID":"0ab774a9-c389-4b5e-ae95-3ed75b46f741","Type":"ContainerStarted","Data":"aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c"} Jul 28 20:55:14.613108 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:14.612648 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" event={"ID":"0ab774a9-c389-4b5e-ae95-3ed75b46f741","Type":"ContainerStarted","Data":"ca8cdec49dad6572e5be41a7987845f1c0e8db4da9bc1ba8cf511aa10d1be987"} Jul 28 20:55:14.634554 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:14.634504 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" podStartSLOduration=1.63448801 podStartE2EDuration="1.63448801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:14.630588951 +0000 UTC m=+3342.074899951" watchObservedRunningTime="2026-07-28 20:55:14.63448801 +0000 UTC m=+3342.078799096" Jul 28 20:55:14.755815 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:14.755781 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:14.759902 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:14.759879 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:15.618160 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:15.618129 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:15.619219 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:15.619200 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:19.517891 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.517857 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6"] Jul 28 20:55:19.521760 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.521738 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.524222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.524158 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d6a3fd-dockercfg-dz82n\"" Jul 28 20:55:19.524222 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.524165 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d6a3fd\"" Jul 28 20:55:19.531239 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.531212 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6"] Jul 28 20:55:19.540897 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.540858 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk"] Jul 28 20:55:19.544636 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.544608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.556824 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.556799 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk"] Jul 28 20:55:19.660148 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.660114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc212fd-0985-4b92-8c75-8c6da67c7add-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.660343 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.660152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slhrt\" (UniqueName: \"kubernetes.io/projected/1cc212fd-0985-4b92-8c75-8c6da67c7add-kube-api-access-slhrt\") pod \"payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.660343 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.660213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62265ad6-038a-4f50-8dc5-61ad7d929665-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.660343 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.660245 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8z5w\" (UniqueName: \"kubernetes.io/projected/62265ad6-038a-4f50-8dc5-61ad7d929665-kube-api-access-f8z5w\") pod \"payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.761314 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.761278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62265ad6-038a-4f50-8dc5-61ad7d929665-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.761522 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.761354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8z5w\" (UniqueName: \"kubernetes.io/projected/62265ad6-038a-4f50-8dc5-61ad7d929665-kube-api-access-f8z5w\") pod \"payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.761522 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.761429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc212fd-0985-4b92-8c75-8c6da67c7add-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.761522 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.761451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slhrt\" (UniqueName: \"kubernetes.io/projected/1cc212fd-0985-4b92-8c75-8c6da67c7add-kube-api-access-slhrt\") pod \"payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.762103 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.762077 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62265ad6-038a-4f50-8dc5-61ad7d929665-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.762245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.762224 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc212fd-0985-4b92-8c75-8c6da67c7add-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.768624 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.768564 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8z5w\" (UniqueName: \"kubernetes.io/projected/62265ad6-038a-4f50-8dc5-61ad7d929665-kube-api-access-f8z5w\") pod \"payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.768624 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.768588 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slhrt\" (UniqueName: \"kubernetes.io/projected/1cc212fd-0985-4b92-8c75-8c6da67c7add-kube-api-access-slhrt\") pod \"payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.834684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.834647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:19.857918 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.857892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:19.988767 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:19.988734 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6"] Jul 28 20:55:20.014764 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.014737 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk"] Jul 28 20:55:20.017666 ip-10-0-136-255 kubenswrapper[2581]: W0728 20:55:20.017636 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc212fd_0985_4b92_8c75_8c6da67c7add.slice/crio-544ff9640090331129c9faba1e1400d9afa9218f7a02a6a5b4d9845edc050f99 WatchSource:0}: Error finding container 544ff9640090331129c9faba1e1400d9afa9218f7a02a6a5b4d9845edc050f99: Status 404 returned error can't find the container with id 544ff9640090331129c9faba1e1400d9afa9218f7a02a6a5b4d9845edc050f99 Jul 28 20:55:20.650964 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.650932 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" event={"ID":"62265ad6-038a-4f50-8dc5-61ad7d929665","Type":"ContainerStarted","Data":"e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a"} Jul 28 20:55:20.651459 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.650995 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:20.651459 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.651022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" event={"ID":"62265ad6-038a-4f50-8dc5-61ad7d929665","Type":"ContainerStarted","Data":"b4377d3881bad5f09f436b5c2b840601eb6d24481dad60596f7e074fc403c933"} Jul 28 20:55:20.652473 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.652448 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" event={"ID":"1cc212fd-0985-4b92-8c75-8c6da67c7add","Type":"ContainerStarted","Data":"168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a"} Jul 28 20:55:20.652588 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.652500 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" event={"ID":"1cc212fd-0985-4b92-8c75-8c6da67c7add","Type":"ContainerStarted","Data":"544ff9640090331129c9faba1e1400d9afa9218f7a02a6a5b4d9845edc050f99"} Jul 28 20:55:20.652588 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.652546 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:20.667664 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.667614 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" podStartSLOduration=1.667600995 podStartE2EDuration="1.667600995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:20.665676893 +0000 UTC m=+3348.109987909" watchObservedRunningTime="2026-07-28 20:55:20.667600995 +0000 UTC m=+3348.111912011" Jul 28 20:55:20.682445 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:20.682394 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" podStartSLOduration=1.682377521 podStartE2EDuration="1.682377521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:55:20.679715644 +0000 UTC m=+3348.124026661" watchObservedRunningTime="2026-07-28 20:55:20.682377521 +0000 UTC m=+3348.126688543" Jul 28 20:55:26.487483 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:26.487445 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:55:31.667392 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:31.667361 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:31.667834 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:31.667434 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:35.768596 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:35.768557 2581 generic.go:356] "Generic (PLEG): container finished" podID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerID="aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c" exitCode=0 Jul 28 20:55:35.769124 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:35.768636 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" event={"ID":"0ab774a9-c389-4b5e-ae95-3ed75b46f741","Type":"ContainerDied","Data":"aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c"} Jul 28 20:55:35.769209 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:35.769182 2581 scope.go:117] "RemoveContainer" containerID="aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c" Jul 28 20:55:35.770427 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:35.770403 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:55:35.770499 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:35.770466 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:55:35.770499 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:35.770496 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 20:55:36.776531 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:36.776495 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" event={"ID":"0ab774a9-c389-4b5e-ae95-3ed75b46f741","Type":"ContainerStarted","Data":"9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c"} Jul 28 20:55:37.754774 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:37.754739 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:37.754997 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:37.754975 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:55:37.755045 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:37.755023 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 28 20:55:38.755773 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:38.755733 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:55:38.756178 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:38.755801 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 28 20:55:39.755287 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:39.755246 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Jul 28 20:55:39.755540 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:39.755303 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Jul 28 20:55:40.759300 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:40.759267 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:40.759698 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:40.759573 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:40.760192 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:40.760171 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:55:49.269784 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.269754 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6"] Jul 28 20:55:49.270281 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.269936 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" podUID="62265ad6-038a-4f50-8dc5-61ad7d929665" containerName="payload-processing" containerID="cri-o://e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a" gracePeriod=30 Jul 28 20:55:49.289624 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.289596 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk"] Jul 28 20:55:49.289866 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.289839 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" podUID="1cc212fd-0985-4b92-8c75-8c6da67c7add" containerName="payload-pre-processing" containerID="cri-o://168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a" gracePeriod=30 Jul 28 20:55:49.585703 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.585678 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:49.589017 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.588995 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:49.644680 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.644621 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slhrt\" (UniqueName: \"kubernetes.io/projected/1cc212fd-0985-4b92-8c75-8c6da67c7add-kube-api-access-slhrt\") pod \"1cc212fd-0985-4b92-8c75-8c6da67c7add\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " Jul 28 20:55:49.644861 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.644694 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62265ad6-038a-4f50-8dc5-61ad7d929665-plugins-config-volume\") pod \"62265ad6-038a-4f50-8dc5-61ad7d929665\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " Jul 28 20:55:49.644861 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.644728 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc212fd-0985-4b92-8c75-8c6da67c7add-plugins-config-volume\") pod \"1cc212fd-0985-4b92-8c75-8c6da67c7add\" (UID: \"1cc212fd-0985-4b92-8c75-8c6da67c7add\") " Jul 28 20:55:49.644861 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.644771 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8z5w\" (UniqueName: \"kubernetes.io/projected/62265ad6-038a-4f50-8dc5-61ad7d929665-kube-api-access-f8z5w\") pod \"62265ad6-038a-4f50-8dc5-61ad7d929665\" (UID: \"62265ad6-038a-4f50-8dc5-61ad7d929665\") " Jul 28 20:55:49.646392 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.646348 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc212fd-0985-4b92-8c75-8c6da67c7add-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cc212fd-0985-4b92-8c75-8c6da67c7add" (UID: "1cc212fd-0985-4b92-8c75-8c6da67c7add"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:55:49.646524 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.646442 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62265ad6-038a-4f50-8dc5-61ad7d929665-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62265ad6-038a-4f50-8dc5-61ad7d929665" (UID: "62265ad6-038a-4f50-8dc5-61ad7d929665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:55:49.647367 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.647297 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc212fd-0985-4b92-8c75-8c6da67c7add-kube-api-access-slhrt" (OuterVolumeSpecName: "kube-api-access-slhrt") pod "1cc212fd-0985-4b92-8c75-8c6da67c7add" (UID: "1cc212fd-0985-4b92-8c75-8c6da67c7add"). InnerVolumeSpecName "kube-api-access-slhrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:55:49.648913 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.648805 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62265ad6-038a-4f50-8dc5-61ad7d929665-kube-api-access-f8z5w" (OuterVolumeSpecName: "kube-api-access-f8z5w") pod "62265ad6-038a-4f50-8dc5-61ad7d929665" (UID: "62265ad6-038a-4f50-8dc5-61ad7d929665"). InnerVolumeSpecName "kube-api-access-f8z5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:55:49.746412 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.746380 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slhrt\" (UniqueName: \"kubernetes.io/projected/1cc212fd-0985-4b92-8c75-8c6da67c7add-kube-api-access-slhrt\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:55:49.746412 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.746408 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62265ad6-038a-4f50-8dc5-61ad7d929665-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:55:49.746412 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.746417 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc212fd-0985-4b92-8c75-8c6da67c7add-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:55:49.746665 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.746427 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8z5w\" (UniqueName: \"kubernetes.io/projected/62265ad6-038a-4f50-8dc5-61ad7d929665-kube-api-access-f8z5w\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:55:49.863644 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.863557 2581 generic.go:356] "Generic (PLEG): container finished" podID="62265ad6-038a-4f50-8dc5-61ad7d929665" containerID="e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a" exitCode=0 Jul 28 20:55:49.863644 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.863609 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" Jul 28 20:55:49.863859 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.863641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" event={"ID":"62265ad6-038a-4f50-8dc5-61ad7d929665","Type":"ContainerDied","Data":"e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a"} Jul 28 20:55:49.863859 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.863723 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6" event={"ID":"62265ad6-038a-4f50-8dc5-61ad7d929665","Type":"ContainerDied","Data":"b4377d3881bad5f09f436b5c2b840601eb6d24481dad60596f7e074fc403c933"} Jul 28 20:55:49.863859 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.863750 2581 scope.go:117] "RemoveContainer" containerID="e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a" Jul 28 20:55:49.864973 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.864948 2581 generic.go:356] "Generic (PLEG): container finished" podID="1cc212fd-0985-4b92-8c75-8c6da67c7add" containerID="168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a" exitCode=0 Jul 28 20:55:49.865127 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.865019 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" Jul 28 20:55:49.865127 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.865028 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" event={"ID":"1cc212fd-0985-4b92-8c75-8c6da67c7add","Type":"ContainerDied","Data":"168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a"} Jul 28 20:55:49.865127 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.865068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk" event={"ID":"1cc212fd-0985-4b92-8c75-8c6da67c7add","Type":"ContainerDied","Data":"544ff9640090331129c9faba1e1400d9afa9218f7a02a6a5b4d9845edc050f99"} Jul 28 20:55:49.874627 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.874603 2581 scope.go:117] "RemoveContainer" containerID="e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a" Jul 28 20:55:49.874932 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:55:49.874906 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a\": container with ID starting with e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a not found: ID does not exist" containerID="e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a" Jul 28 20:55:49.875051 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.874945 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a"} err="failed to get container status \"e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a\": rpc error: code = NotFound desc = could not find container \"e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a\": container with ID starting with e5664cd3c828638de36fb8e44d3961b4a30d93186e4f1f57030fa8bbba61822a not found: ID does not exist" Jul 28 20:55:49.875051 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.874967 2581 scope.go:117] "RemoveContainer" containerID="168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a" Jul 28 20:55:49.884272 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.884254 2581 scope.go:117] "RemoveContainer" containerID="168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a" Jul 28 20:55:49.884548 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:55:49.884528 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a\": container with ID starting with 168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a not found: ID does not exist" containerID="168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a" Jul 28 20:55:49.884619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.884555 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a"} err="failed to get container status \"168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a\": rpc error: code = NotFound desc = could not find container \"168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a\": container with ID starting with 168d31aa03238eee388ee016d2507d3e9d9efdf8813cce08834959984090f84a not found: ID does not exist" Jul 28 20:55:49.899472 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.899439 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk"] Jul 28 20:55:49.905297 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.905275 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d6a3fd-847d9cdfb8-hd7qk"] Jul 28 20:55:49.934136 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.934085 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6"] Jul 28 20:55:49.937372 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:49.937343 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d6a3fd-699b64d996-kl7w6"] Jul 28 20:55:51.153591 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:51.153557 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc212fd-0985-4b92-8c75-8c6da67c7add" path="/var/lib/kubelet/pods/1cc212fd-0985-4b92-8c75-8c6da67c7add/volumes" Jul 28 20:55:51.153967 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:51.153933 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62265ad6-038a-4f50-8dc5-61ad7d929665" path="/var/lib/kubelet/pods/62265ad6-038a-4f50-8dc5-61ad7d929665/volumes" Jul 28 20:55:53.798594 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:53.798554 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:55:56.845119 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:56.845084 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r"] Jul 28 20:55:56.845610 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:55:56.845393 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" containerID="cri-o://9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c" gracePeriod=30 Jul 28 20:56:02.105608 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.105545 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:56:02.181108 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181074 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-data\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181270 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181118 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-token\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181270 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181182 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istiod-ca-cert\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181270 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181245 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9cf8\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-kube-api-access-s9cf8\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181443 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181307 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-envoy\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181443 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181361 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-socket\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181443 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181390 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-podinfo\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181443 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181421 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-certs\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181649 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181449 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-credential-socket\") pod \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\" (UID: \"0ab774a9-c389-4b5e-ae95-3ed75b46f741\") " Jul 28 20:56:02.181710 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181638 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:56:02.181710 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181661 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:56:02.181850 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181833 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:56:02.181911 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181886 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istiod-ca-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.181969 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181909 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.181969 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181890 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:56:02.182072 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.181985 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:56:02.183591 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.183559 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:56:02.183705 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.183636 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:56:02.183833 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.183699 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:56:02.183833 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.183720 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-kube-api-access-s9cf8" (OuterVolumeSpecName: "kube-api-access-s9cf8") pod "0ab774a9-c389-4b5e-ae95-3ed75b46f741" (UID: "0ab774a9-c389-4b5e-ae95-3ed75b46f741"). InnerVolumeSpecName "kube-api-access-s9cf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:56:02.282682 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.282647 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9cf8\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-kube-api-access-s9cf8\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.282682 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.282678 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-envoy\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.282682 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.282689 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-podinfo\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.282921 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.282697 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-workload-certs\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.282921 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.282706 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-credential-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.282921 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.282714 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-data\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.282921 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.282722 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab774a9-c389-4b5e-ae95-3ed75b46f741-istio-token\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:02.946450 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.946415 2581 generic.go:356] "Generic (PLEG): container finished" podID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerID="9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c" exitCode=0 Jul 28 20:56:02.946450 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.946452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" event={"ID":"0ab774a9-c389-4b5e-ae95-3ed75b46f741","Type":"ContainerDied","Data":"9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c"} Jul 28 20:56:02.946721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.946470 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" Jul 28 20:56:02.946721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.946495 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r" event={"ID":"0ab774a9-c389-4b5e-ae95-3ed75b46f741","Type":"ContainerDied","Data":"ca8cdec49dad6572e5be41a7987845f1c0e8db4da9bc1ba8cf511aa10d1be987"} Jul 28 20:56:02.946721 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.946518 2581 scope.go:117] "RemoveContainer" containerID="9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c" Jul 28 20:56:02.957245 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.957224 2581 scope.go:117] "RemoveContainer" containerID="aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c" Jul 28 20:56:02.965915 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.965747 2581 scope.go:117] "RemoveContainer" containerID="9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c" Jul 28 20:56:02.966011 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:56:02.965993 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c\": container with ID starting with 9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c not found: ID does not exist" containerID="9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c" Jul 28 20:56:02.966056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.966020 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c"} err="failed to get container status \"9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c\": rpc error: code = NotFound desc = could not find container \"9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c\": container with ID starting with 9a1349e1cd6ac7268a79a5f09fa3d352830c59e8b0fa4d58587ac5baced49e7c not found: ID does not exist" Jul 28 20:56:02.966056 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.966038 2581 scope.go:117] "RemoveContainer" containerID="aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c" Jul 28 20:56:02.966240 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:56:02.966222 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c\": container with ID starting with aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c not found: ID does not exist" containerID="aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c" Jul 28 20:56:02.966296 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.966250 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c"} err="failed to get container status \"aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c\": rpc error: code = NotFound desc = could not find container \"aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c\": container with ID starting with aaa070a7e0f52ad54369f99a1e0dfd708d5558916a3e012a70a9c633c590a10c not found: ID does not exist" Jul 28 20:56:02.973753 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.973730 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r"] Jul 28 20:56:02.978377 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:02.978356 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-d6a3fd-openshift-default-86c5f5dd44-rsx5r"] Jul 28 20:56:03.154037 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:03.154005 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" path="/var/lib/kubelet/pods/0ab774a9-c389-4b5e-ae95-3ed75b46f741/volumes" Jul 28 20:56:14.889285 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:14.889247 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:56:19.276367 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.276305 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6"] Jul 28 20:56:19.276841 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.276587 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" podUID="b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" containerName="payload-processing" containerID="cri-o://c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8" gracePeriod=30 Jul 28 20:56:19.285544 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.285059 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88"] Jul 28 20:56:19.285544 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.285278 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" podUID="939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" containerName="payload-pre-processing" containerID="cri-o://a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6" gracePeriod=30 Jul 28 20:56:19.600873 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.600853 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:56:19.603982 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.603962 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:56:19.756465 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.756434 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-plugins-config-volume\") pod \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " Jul 28 20:56:19.756661 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.756578 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x546l\" (UniqueName: \"kubernetes.io/projected/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-kube-api-access-x546l\") pod \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\" (UID: \"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387\") " Jul 28 20:56:19.756661 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.756602 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-plugins-config-volume\") pod \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " Jul 28 20:56:19.756661 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.756620 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcn2j\" (UniqueName: \"kubernetes.io/projected/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-kube-api-access-xcn2j\") pod \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\" (UID: \"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36\") " Jul 28 20:56:19.756896 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.756871 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" (UID: "b19fdb3a-f626-4d8c-b7d9-daeefa9c0387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:56:19.756959 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.756940 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" (UID: "939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:56:19.758671 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.758635 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-kube-api-access-xcn2j" (OuterVolumeSpecName: "kube-api-access-xcn2j") pod "939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" (UID: "939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36"). InnerVolumeSpecName "kube-api-access-xcn2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:56:19.758781 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.758767 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-kube-api-access-x546l" (OuterVolumeSpecName: "kube-api-access-x546l") pod "b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" (UID: "b19fdb3a-f626-4d8c-b7d9-daeefa9c0387"). InnerVolumeSpecName "kube-api-access-x546l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:56:19.858199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.858122 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x546l\" (UniqueName: \"kubernetes.io/projected/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-kube-api-access-x546l\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:19.858199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.858151 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:19.858199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.858160 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcn2j\" (UniqueName: \"kubernetes.io/projected/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36-kube-api-access-xcn2j\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:19.858199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:19.858169 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:56:20.048702 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.048666 2581 generic.go:356] "Generic (PLEG): container finished" podID="b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" containerID="c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8" exitCode=0 Jul 28 20:56:20.048897 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.048733 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" Jul 28 20:56:20.048897 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.048753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" event={"ID":"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387","Type":"ContainerDied","Data":"c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8"} Jul 28 20:56:20.048897 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.048834 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6" event={"ID":"b19fdb3a-f626-4d8c-b7d9-daeefa9c0387","Type":"ContainerDied","Data":"f19fc99585656b1cfd02f8635d2225ef142d3482ad4bad2396793be0b7737726"} Jul 28 20:56:20.048897 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.048858 2581 scope.go:117] "RemoveContainer" containerID="c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8" Jul 28 20:56:20.050164 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.050143 2581 generic.go:356] "Generic (PLEG): container finished" podID="939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" containerID="a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6" exitCode=0 Jul 28 20:56:20.050260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.050205 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" Jul 28 20:56:20.050260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.050218 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" event={"ID":"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36","Type":"ContainerDied","Data":"a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6"} Jul 28 20:56:20.050260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.050248 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88" event={"ID":"939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36","Type":"ContainerDied","Data":"8d08438b37ca5c4622aa6fcc5489102a472a191de84cae56b8acb5b224011a69"} Jul 28 20:56:20.058077 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.058037 2581 scope.go:117] "RemoveContainer" containerID="c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8" Jul 28 20:56:20.058385 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:56:20.058306 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8\": container with ID starting with c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8 not found: ID does not exist" containerID="c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8" Jul 28 20:56:20.058385 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.058354 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8"} err="failed to get container status \"c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8\": rpc error: code = NotFound desc = could not find container \"c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8\": container with ID starting with c263892815d0d7845b7f90dec8899d40af24e05e9cbad00b248c7e692ee3b7e8 not found: ID does not exist" Jul 28 20:56:20.058385 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.058372 2581 scope.go:117] "RemoveContainer" containerID="a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6" Jul 28 20:56:20.066797 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.066782 2581 scope.go:117] "RemoveContainer" containerID="a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6" Jul 28 20:56:20.067068 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:56:20.067050 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6\": container with ID starting with a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6 not found: ID does not exist" containerID="a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6" Jul 28 20:56:20.067125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.067073 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6"} err="failed to get container status \"a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6\": rpc error: code = NotFound desc = could not find container \"a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6\": container with ID starting with a0458e3107a19d4361bc40d486e6036f224848b2c680039b7e9c23a3733e21f6 not found: ID does not exist" Jul 28 20:56:20.094066 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.094042 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88"] Jul 28 20:56:20.097677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.097655 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dfc5b-6b6956c6d4-pmm88"] Jul 28 20:56:20.120437 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.120376 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6"] Jul 28 20:56:20.129910 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:20.129892 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dfc5b-7bc4cc6994-zzbn6"] Jul 28 20:56:21.151942 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:21.151909 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" path="/var/lib/kubelet/pods/939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36/volumes" Jul 28 20:56:21.152308 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:21.152261 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" path="/var/lib/kubelet/pods/b19fdb3a-f626-4d8c-b7d9-daeefa9c0387/volumes" Jul 28 20:56:30.593044 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:56:30.593007 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:57:15.501040 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.500999 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc"] Jul 28 20:57:15.501637 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.501277 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" podUID="7925da09-1920-463d-ab3c-4c48782cec9d" containerName="payload-processing" containerID="cri-o://ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967" gracePeriod=30 Jul 28 20:57:15.531733 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.531707 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c"] Jul 28 20:57:15.531975 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.531953 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" podUID="427171fc-9f77-46d6-a210-63f7b27504d3" containerName="payload-pre-processing" containerID="cri-o://d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8" gracePeriod=30 Jul 28 20:57:15.817138 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.817115 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:57:15.818376 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.818356 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbftk\" (UniqueName: \"kubernetes.io/projected/7925da09-1920-463d-ab3c-4c48782cec9d-kube-api-access-wbftk\") pod \"7925da09-1920-463d-ab3c-4c48782cec9d\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " Jul 28 20:57:15.818491 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.818419 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7925da09-1920-463d-ab3c-4c48782cec9d-plugins-config-volume\") pod \"7925da09-1920-463d-ab3c-4c48782cec9d\" (UID: \"7925da09-1920-463d-ab3c-4c48782cec9d\") " Jul 28 20:57:15.818769 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.818746 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7925da09-1920-463d-ab3c-4c48782cec9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7925da09-1920-463d-ab3c-4c48782cec9d" (UID: "7925da09-1920-463d-ab3c-4c48782cec9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:57:15.820260 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.820240 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:57:15.820386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.820366 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7925da09-1920-463d-ab3c-4c48782cec9d-kube-api-access-wbftk" (OuterVolumeSpecName: "kube-api-access-wbftk") pod "7925da09-1920-463d-ab3c-4c48782cec9d" (UID: "7925da09-1920-463d-ab3c-4c48782cec9d"). InnerVolumeSpecName "kube-api-access-wbftk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:15.919337 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.919300 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427171fc-9f77-46d6-a210-63f7b27504d3-plugins-config-volume\") pod \"427171fc-9f77-46d6-a210-63f7b27504d3\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " Jul 28 20:57:15.919607 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.919555 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5c2g\" (UniqueName: \"kubernetes.io/projected/427171fc-9f77-46d6-a210-63f7b27504d3-kube-api-access-z5c2g\") pod \"427171fc-9f77-46d6-a210-63f7b27504d3\" (UID: \"427171fc-9f77-46d6-a210-63f7b27504d3\") " Jul 28 20:57:15.919726 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.919703 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427171fc-9f77-46d6-a210-63f7b27504d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "427171fc-9f77-46d6-a210-63f7b27504d3" (UID: "427171fc-9f77-46d6-a210-63f7b27504d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:57:15.919889 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.919874 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7925da09-1920-463d-ab3c-4c48782cec9d-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:15.919889 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.919888 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427171fc-9f77-46d6-a210-63f7b27504d3-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:15.920041 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.919897 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbftk\" (UniqueName: \"kubernetes.io/projected/7925da09-1920-463d-ab3c-4c48782cec9d-kube-api-access-wbftk\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:15.921594 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:15.921569 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427171fc-9f77-46d6-a210-63f7b27504d3-kube-api-access-z5c2g" (OuterVolumeSpecName: "kube-api-access-z5c2g") pod "427171fc-9f77-46d6-a210-63f7b27504d3" (UID: "427171fc-9f77-46d6-a210-63f7b27504d3"). InnerVolumeSpecName "kube-api-access-z5c2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:16.020600 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.020570 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5c2g\" (UniqueName: \"kubernetes.io/projected/427171fc-9f77-46d6-a210-63f7b27504d3-kube-api-access-z5c2g\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:16.396189 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.396157 2581 generic.go:356] "Generic (PLEG): container finished" podID="427171fc-9f77-46d6-a210-63f7b27504d3" containerID="d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8" exitCode=0 Jul 28 20:57:16.396413 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.396204 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" Jul 28 20:57:16.396413 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.396244 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" event={"ID":"427171fc-9f77-46d6-a210-63f7b27504d3","Type":"ContainerDied","Data":"d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8"} Jul 28 20:57:16.396413 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.396301 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c" event={"ID":"427171fc-9f77-46d6-a210-63f7b27504d3","Type":"ContainerDied","Data":"040460b58ddac04d33d6e21f1b263ded5fc609f15baf777feb3d55be1c6f37eb"} Jul 28 20:57:16.396413 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.396317 2581 scope.go:117] "RemoveContainer" containerID="d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8" Jul 28 20:57:16.397506 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.397475 2581 generic.go:356] "Generic (PLEG): container finished" podID="7925da09-1920-463d-ab3c-4c48782cec9d" containerID="ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967" exitCode=0 Jul 28 20:57:16.397604 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.397511 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" event={"ID":"7925da09-1920-463d-ab3c-4c48782cec9d","Type":"ContainerDied","Data":"ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967"} Jul 28 20:57:16.397604 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.397530 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" Jul 28 20:57:16.397604 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.397560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc" event={"ID":"7925da09-1920-463d-ab3c-4c48782cec9d","Type":"ContainerDied","Data":"ba81b94c29645a00a6323fe59ae31f489517a3a4a2221c83239230410baffdad"} Jul 28 20:57:16.405715 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.405697 2581 scope.go:117] "RemoveContainer" containerID="d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8" Jul 28 20:57:16.405992 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:16.405965 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8\": container with ID starting with d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8 not found: ID does not exist" containerID="d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8" Jul 28 20:57:16.406092 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.406005 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8"} err="failed to get container status \"d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8\": rpc error: code = NotFound desc = could not find container \"d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8\": container with ID starting with d31c571f264508900d116e08e1d799b925ff2d529008fff2749076942c9cdcf8 not found: ID does not exist" Jul 28 20:57:16.406092 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.406029 2581 scope.go:117] "RemoveContainer" containerID="ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967" Jul 28 20:57:16.414304 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.414284 2581 scope.go:117] "RemoveContainer" containerID="ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967" Jul 28 20:57:16.414625 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:16.414603 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967\": container with ID starting with ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967 not found: ID does not exist" containerID="ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967" Jul 28 20:57:16.414702 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.414629 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967"} err="failed to get container status \"ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967\": rpc error: code = NotFound desc = could not find container \"ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967\": container with ID starting with ab0f9bde0867da27bcb3810aaa9229a488179049f754c8b757d6c0bc2087b967 not found: ID does not exist" Jul 28 20:57:16.428596 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.428572 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc"] Jul 28 20:57:16.435765 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.435746 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ac855-56b668cb54-7xdzc"] Jul 28 20:57:16.449309 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.449290 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c"] Jul 28 20:57:16.455518 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:16.455499 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ac855-bc84b9d64-s7q7c"] Jul 28 20:57:17.153028 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:17.152988 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="427171fc-9f77-46d6-a210-63f7b27504d3" path="/var/lib/kubelet/pods/427171fc-9f77-46d6-a210-63f7b27504d3/volumes" Jul 28 20:57:17.153548 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:17.153528 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7925da09-1920-463d-ab3c-4c48782cec9d" path="/var/lib/kubelet/pods/7925da09-1920-463d-ab3c-4c48782cec9d/volumes" Jul 28 20:57:18.780130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:18.780087 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:57:30.659877 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:30.659845 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j"] Jul 28 20:57:30.660459 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:30.660049 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" podUID="4186e0f0-0d24-44bc-bf46-dbae729e37ae" containerName="payload-processing" containerID="cri-o://67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356" gracePeriod=30 Jul 28 20:57:30.670351 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:30.670304 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg"] Jul 28 20:57:30.670614 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:30.670561 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" podUID="558c5b3c-542d-46ea-8b9d-bd3df13777bd" containerName="payload-pre-processing" containerID="cri-o://f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f" gracePeriod=30 Jul 28 20:57:30.948125 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:30.948103 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:57:30.950998 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:30.950979 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:57:31.047962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.047935 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzwtv\" (UniqueName: \"kubernetes.io/projected/4186e0f0-0d24-44bc-bf46-dbae729e37ae-kube-api-access-jzwtv\") pod \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " Jul 28 20:57:31.048142 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.047969 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558c5b3c-542d-46ea-8b9d-bd3df13777bd-plugins-config-volume\") pod \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " Jul 28 20:57:31.048142 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.047996 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxm2t\" (UniqueName: \"kubernetes.io/projected/558c5b3c-542d-46ea-8b9d-bd3df13777bd-kube-api-access-kxm2t\") pod \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\" (UID: \"558c5b3c-542d-46ea-8b9d-bd3df13777bd\") " Jul 28 20:57:31.048283 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.048176 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4186e0f0-0d24-44bc-bf46-dbae729e37ae-plugins-config-volume\") pod \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\" (UID: \"4186e0f0-0d24-44bc-bf46-dbae729e37ae\") " Jul 28 20:57:31.048498 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.048400 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558c5b3c-542d-46ea-8b9d-bd3df13777bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "558c5b3c-542d-46ea-8b9d-bd3df13777bd" (UID: "558c5b3c-542d-46ea-8b9d-bd3df13777bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:57:31.048595 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.048575 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4186e0f0-0d24-44bc-bf46-dbae729e37ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4186e0f0-0d24-44bc-bf46-dbae729e37ae" (UID: "4186e0f0-0d24-44bc-bf46-dbae729e37ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:57:31.050070 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.050052 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4186e0f0-0d24-44bc-bf46-dbae729e37ae-kube-api-access-jzwtv" (OuterVolumeSpecName: "kube-api-access-jzwtv") pod "4186e0f0-0d24-44bc-bf46-dbae729e37ae" (UID: "4186e0f0-0d24-44bc-bf46-dbae729e37ae"). InnerVolumeSpecName "kube-api-access-jzwtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:31.050158 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.050140 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558c5b3c-542d-46ea-8b9d-bd3df13777bd-kube-api-access-kxm2t" (OuterVolumeSpecName: "kube-api-access-kxm2t") pod "558c5b3c-542d-46ea-8b9d-bd3df13777bd" (UID: "558c5b3c-542d-46ea-8b9d-bd3df13777bd"). InnerVolumeSpecName "kube-api-access-kxm2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:31.149631 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.149604 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwtv\" (UniqueName: \"kubernetes.io/projected/4186e0f0-0d24-44bc-bf46-dbae729e37ae-kube-api-access-jzwtv\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:31.149631 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.149628 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558c5b3c-542d-46ea-8b9d-bd3df13777bd-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:31.149631 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.149637 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxm2t\" (UniqueName: \"kubernetes.io/projected/558c5b3c-542d-46ea-8b9d-bd3df13777bd-kube-api-access-kxm2t\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:31.149848 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.149647 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4186e0f0-0d24-44bc-bf46-dbae729e37ae-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:31.487978 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.487948 2581 generic.go:356] "Generic (PLEG): container finished" podID="4186e0f0-0d24-44bc-bf46-dbae729e37ae" containerID="67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356" exitCode=0 Jul 28 20:57:31.488187 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.487998 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" Jul 28 20:57:31.488187 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.488030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" event={"ID":"4186e0f0-0d24-44bc-bf46-dbae729e37ae","Type":"ContainerDied","Data":"67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356"} Jul 28 20:57:31.488187 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.488081 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j" event={"ID":"4186e0f0-0d24-44bc-bf46-dbae729e37ae","Type":"ContainerDied","Data":"39961f3e206f044e3932a23c23c8f6eedb87afad75b3f96c0e1db587e09eb8a7"} Jul 28 20:57:31.488187 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.488105 2581 scope.go:117] "RemoveContainer" containerID="67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356" Jul 28 20:57:31.489500 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.489475 2581 generic.go:356] "Generic (PLEG): container finished" podID="558c5b3c-542d-46ea-8b9d-bd3df13777bd" containerID="f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f" exitCode=0 Jul 28 20:57:31.489618 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.489503 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" event={"ID":"558c5b3c-542d-46ea-8b9d-bd3df13777bd","Type":"ContainerDied","Data":"f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f"} Jul 28 20:57:31.489618 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.489532 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" event={"ID":"558c5b3c-542d-46ea-8b9d-bd3df13777bd","Type":"ContainerDied","Data":"4291b80ccb03a6961a366d5ec32fe4bdda315da9ba8bc7dc96296182e2289be8"} Jul 28 20:57:31.489618 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.489552 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg" Jul 28 20:57:31.497567 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.497548 2581 scope.go:117] "RemoveContainer" containerID="67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356" Jul 28 20:57:31.497808 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:31.497781 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356\": container with ID starting with 67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356 not found: ID does not exist" containerID="67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356" Jul 28 20:57:31.497858 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.497815 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356"} err="failed to get container status \"67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356\": rpc error: code = NotFound desc = could not find container \"67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356\": container with ID starting with 67a7b83f6885c3bc572c416cceda9a2e8572ab43112756534180feae06664356 not found: ID does not exist" Jul 28 20:57:31.497858 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.497830 2581 scope.go:117] "RemoveContainer" containerID="f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f" Jul 28 20:57:31.506192 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.506175 2581 scope.go:117] "RemoveContainer" containerID="f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f" Jul 28 20:57:31.506499 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:31.506478 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f\": container with ID starting with f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f not found: ID does not exist" containerID="f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f" Jul 28 20:57:31.506566 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.506505 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f"} err="failed to get container status \"f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f\": rpc error: code = NotFound desc = could not find container \"f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f\": container with ID starting with f2be3b8cd3cb4defbcc9b1d93439d2fab21af0876330996dfcc4bd5aab08ce8f not found: ID does not exist" Jul 28 20:57:31.506620 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.506581 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j"] Jul 28 20:57:31.510415 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.510394 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1b6922-7b57bc4bfd-lgf4j"] Jul 28 20:57:31.520257 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.520239 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg"] Jul 28 20:57:31.525700 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:31.525677 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1b6922-57949f4856-7bvhg"] Jul 28 20:57:33.153189 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:33.153152 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4186e0f0-0d24-44bc-bf46-dbae729e37ae" path="/var/lib/kubelet/pods/4186e0f0-0d24-44bc-bf46-dbae729e37ae/volumes" Jul 28 20:57:33.153564 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:33.153525 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558c5b3c-542d-46ea-8b9d-bd3df13777bd" path="/var/lib/kubelet/pods/558c5b3c-542d-46ea-8b9d-bd3df13777bd/volumes" Jul 28 20:57:35.587537 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:35.587503 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:57:43.569735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.569703 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2"] Jul 28 20:57:43.572562 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.572538 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-c0e826-29754525-c6xw2"] Jul 28 20:57:43.601198 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.601168 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq"] Jul 28 20:57:43.601678 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.601623 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" podUID="9ef1f1de-b499-4926-af32-ffa76297d75e" containerName="payload-processing" containerID="cri-o://4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66" gracePeriod=30 Jul 28 20:57:43.610172 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.610143 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9"] Jul 28 20:57:43.610424 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.610380 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" podUID="78296855-fbc4-4910-ac32-e9deda856149" containerName="payload-pre-processing" containerID="cri-o://1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66" gracePeriod=30 Jul 28 20:57:43.873789 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.873767 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:57:43.903332 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.903294 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:57:43.969340 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.969294 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef1f1de-b499-4926-af32-ffa76297d75e-plugins-config-volume\") pod \"9ef1f1de-b499-4926-af32-ffa76297d75e\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " Jul 28 20:57:43.969479 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.969389 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n9tw\" (UniqueName: \"kubernetes.io/projected/9ef1f1de-b499-4926-af32-ffa76297d75e-kube-api-access-6n9tw\") pod \"9ef1f1de-b499-4926-af32-ffa76297d75e\" (UID: \"9ef1f1de-b499-4926-af32-ffa76297d75e\") " Jul 28 20:57:43.969479 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.969414 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbkgp\" (UniqueName: \"kubernetes.io/projected/78296855-fbc4-4910-ac32-e9deda856149-kube-api-access-bbkgp\") pod \"78296855-fbc4-4910-ac32-e9deda856149\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " Jul 28 20:57:43.969479 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.969443 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78296855-fbc4-4910-ac32-e9deda856149-plugins-config-volume\") pod \"78296855-fbc4-4910-ac32-e9deda856149\" (UID: \"78296855-fbc4-4910-ac32-e9deda856149\") " Jul 28 20:57:43.969792 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.969706 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef1f1de-b499-4926-af32-ffa76297d75e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ef1f1de-b499-4926-af32-ffa76297d75e" (UID: "9ef1f1de-b499-4926-af32-ffa76297d75e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:57:43.969880 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.969855 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78296855-fbc4-4910-ac32-e9deda856149-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78296855-fbc4-4910-ac32-e9deda856149" (UID: "78296855-fbc4-4910-ac32-e9deda856149"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:57:43.971448 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.971425 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef1f1de-b499-4926-af32-ffa76297d75e-kube-api-access-6n9tw" (OuterVolumeSpecName: "kube-api-access-6n9tw") pod "9ef1f1de-b499-4926-af32-ffa76297d75e" (UID: "9ef1f1de-b499-4926-af32-ffa76297d75e"). InnerVolumeSpecName "kube-api-access-6n9tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:43.971540 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:43.971447 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78296855-fbc4-4910-ac32-e9deda856149-kube-api-access-bbkgp" (OuterVolumeSpecName: "kube-api-access-bbkgp") pod "78296855-fbc4-4910-ac32-e9deda856149" (UID: "78296855-fbc4-4910-ac32-e9deda856149"). InnerVolumeSpecName "kube-api-access-bbkgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:44.070173 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.070134 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbkgp\" (UniqueName: \"kubernetes.io/projected/78296855-fbc4-4910-ac32-e9deda856149-kube-api-access-bbkgp\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:44.070173 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.070167 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78296855-fbc4-4910-ac32-e9deda856149-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:44.070386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.070184 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef1f1de-b499-4926-af32-ffa76297d75e-plugins-config-volume\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:44.070386 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.070199 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9tw\" (UniqueName: \"kubernetes.io/projected/9ef1f1de-b499-4926-af32-ffa76297d75e-kube-api-access-6n9tw\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:44.565431 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.565397 2581 generic.go:356] "Generic (PLEG): container finished" podID="78296855-fbc4-4910-ac32-e9deda856149" containerID="1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66" exitCode=0 Jul 28 20:57:44.565619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.565455 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" Jul 28 20:57:44.565619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.565481 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" event={"ID":"78296855-fbc4-4910-ac32-e9deda856149","Type":"ContainerDied","Data":"1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66"} Jul 28 20:57:44.565619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.565531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9" event={"ID":"78296855-fbc4-4910-ac32-e9deda856149","Type":"ContainerDied","Data":"c4f519dfe747aec9146493877ae79c16814e016a29c93d196ec2df052b6d40d0"} Jul 28 20:57:44.565619 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.565561 2581 scope.go:117] "RemoveContainer" containerID="1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66" Jul 28 20:57:44.566901 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.566847 2581 generic.go:356] "Generic (PLEG): container finished" podID="9ef1f1de-b499-4926-af32-ffa76297d75e" containerID="4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66" exitCode=0 Jul 28 20:57:44.566977 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.566896 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" Jul 28 20:57:44.566977 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.566922 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" event={"ID":"9ef1f1de-b499-4926-af32-ffa76297d75e","Type":"ContainerDied","Data":"4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66"} Jul 28 20:57:44.566977 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.566967 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq" event={"ID":"9ef1f1de-b499-4926-af32-ffa76297d75e","Type":"ContainerDied","Data":"2bec423d3fea4f62843cdca16d96378727605c581b5aa037df65513dba11942b"} Jul 28 20:57:44.575701 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.575685 2581 scope.go:117] "RemoveContainer" containerID="1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66" Jul 28 20:57:44.576002 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:44.575980 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66\": container with ID starting with 1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66 not found: ID does not exist" containerID="1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66" Jul 28 20:57:44.576048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.576016 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66"} err="failed to get container status \"1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66\": rpc error: code = NotFound desc = could not find container \"1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66\": container with ID starting with 1d56499d5b7c5a044ffc539b98f205421a74b4741213c5a719749279c69c0b66 not found: ID does not exist" Jul 28 20:57:44.576048 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.576043 2581 scope.go:117] "RemoveContainer" containerID="4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66" Jul 28 20:57:44.584107 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.584091 2581 scope.go:117] "RemoveContainer" containerID="4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66" Jul 28 20:57:44.584360 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:44.584341 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66\": container with ID starting with 4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66 not found: ID does not exist" containerID="4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66" Jul 28 20:57:44.584454 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.584363 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66"} err="failed to get container status \"4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66\": rpc error: code = NotFound desc = could not find container \"4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66\": container with ID starting with 4a7579e3b1a40f83b8633fd15c45bc8c4c6cd01852eae078d5ca1aba799ada66 not found: ID does not exist" Jul 28 20:57:44.590751 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.590731 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9"] Jul 28 20:57:44.594181 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.594160 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-c0e826-85d9c4b4d8-7jlz9"] Jul 28 20:57:44.604870 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.604847 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq"] Jul 28 20:57:44.607108 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:44.607089 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-c0e826-576855b46b-hl2jq"] Jul 28 20:57:45.153951 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:45.153918 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78296855-fbc4-4910-ac32-e9deda856149" path="/var/lib/kubelet/pods/78296855-fbc4-4910-ac32-e9deda856149/volumes" Jul 28 20:57:45.154279 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:45.154266 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef1f1de-b499-4926-af32-ffa76297d75e" path="/var/lib/kubelet/pods/9ef1f1de-b499-4926-af32-ffa76297d75e/volumes" Jul 28 20:57:45.154536 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:45.154524 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59dbe31-75da-44a6-9de6-7360a4ba6e6d" path="/var/lib/kubelet/pods/d59dbe31-75da-44a6-9de6-7360a4ba6e6d/volumes" Jul 28 20:57:47.893108 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:47.893076 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tth4c"] Jul 28 20:57:50.851173 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:50.851144 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s"] Jul 28 20:57:50.851582 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:50.851370 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" containerID="cri-o://33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87" gracePeriod=30 Jul 28 20:57:56.113145 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.113120 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:57:56.278100 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278066 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-data\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278310 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278119 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj25f\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-kube-api-access-dj25f\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278310 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278140 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-credential-socket\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278310 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278158 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-certs\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278310 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278187 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-socket\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278310 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278227 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-token\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278310 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278243 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-envoy\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278317 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istiod-ca-cert\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278371 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-podinfo\") pod \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\" (UID: \"51786bf7-54c3-43fd-8b40-b49fa5b2f280\") " Jul 28 20:57:56.278684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278479 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:57:56.278684 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278631 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:57:56.278879 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278746 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-credential-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.278879 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278770 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-certs\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.278990 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278971 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:57:56.278990 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.278980 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 20:57:56.279136 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.279116 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-data" (OuterVolumeSpecName: "istio-data") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:57:56.280571 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.280544 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-kube-api-access-dj25f" (OuterVolumeSpecName: "kube-api-access-dj25f") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "kube-api-access-dj25f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:56.280677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.280605 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 20:57:56.280847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.280829 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-token" (OuterVolumeSpecName: "istio-token") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 20:57:56.280847 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.280834 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51786bf7-54c3-43fd-8b40-b49fa5b2f280" (UID: "51786bf7-54c3-43fd-8b40-b49fa5b2f280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 20:57:56.379464 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.379417 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-workload-socket\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.379464 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.379451 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-token\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.379464 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.379461 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-envoy\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.379464 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.379469 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istiod-ca-cert\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.379464 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.379476 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-podinfo\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.379769 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.379485 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51786bf7-54c3-43fd-8b40-b49fa5b2f280-istio-data\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.379769 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.379495 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj25f\" (UniqueName: \"kubernetes.io/projected/51786bf7-54c3-43fd-8b40-b49fa5b2f280-kube-api-access-dj25f\") on node \"ip-10-0-136-255.ec2.internal\" DevicePath \"\"" Jul 28 20:57:56.636032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.635936 2581 generic.go:356] "Generic (PLEG): container finished" podID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerID="33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87" exitCode=0 Jul 28 20:57:56.636032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.636002 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" Jul 28 20:57:56.636032 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.636023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" event={"ID":"51786bf7-54c3-43fd-8b40-b49fa5b2f280","Type":"ContainerDied","Data":"33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87"} Jul 28 20:57:56.636289 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.636101 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s" event={"ID":"51786bf7-54c3-43fd-8b40-b49fa5b2f280","Type":"ContainerDied","Data":"e13851097dc43d098e3bcc4334ebaeb2616dce0e03a27f6af47b99c696575f51"} Jul 28 20:57:56.636289 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.636125 2581 scope.go:117] "RemoveContainer" containerID="33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87" Jul 28 20:57:56.646440 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.646420 2581 scope.go:117] "RemoveContainer" containerID="c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4" Jul 28 20:57:56.655384 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.655362 2581 scope.go:117] "RemoveContainer" containerID="33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87" Jul 28 20:57:56.655694 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:56.655674 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87\": container with ID starting with 33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87 not found: ID does not exist" containerID="33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87" Jul 28 20:57:56.655776 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.655709 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87"} err="failed to get container status \"33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87\": rpc error: code = NotFound desc = could not find container \"33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87\": container with ID starting with 33ae2059595bbad63b2c978ad588f9f9205dfbdfeb39969508ca2a66c3fdbc87 not found: ID does not exist" Jul 28 20:57:56.655776 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.655736 2581 scope.go:117] "RemoveContainer" containerID="c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4" Jul 28 20:57:56.656026 ip-10-0-136-255 kubenswrapper[2581]: E0728 20:57:56.656005 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4\": container with ID starting with c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4 not found: ID does not exist" containerID="c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4" Jul 28 20:57:56.656070 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.656031 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4"} err="failed to get container status \"c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4\": rpc error: code = NotFound desc = could not find container \"c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4\": container with ID starting with c06a8324b1b269b51b0ff5c79569199a8886cb986cb8656c74c055390728b0f4 not found: ID does not exist" Jul 28 20:57:56.660840 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.660812 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s"] Jul 28 20:57:56.666068 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:56.666038 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c0e826-openshift-default-66d5bc5476-4xj8s"] Jul 28 20:57:57.153910 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:57.153878 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" path="/var/lib/kubelet/pods/51786bf7-54c3-43fd-8b40-b49fa5b2f280/volumes" Jul 28 20:57:57.868872 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:57.868842 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-5tjgw_3ca056c4-b45b-42e3-80e7-2483d9b2f0d6/manager/0.log" Jul 28 20:57:57.875366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:57.875316 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-5tjgw_3ca056c4-b45b-42e3-80e7-2483d9b2f0d6/copy-manifests/0.log" Jul 28 20:57:58.502102 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:58.502070 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754495-4svqf_552a8af6-9749-4128-bd82-83cacf51515a/cleanup/0.log" Jul 28 20:57:58.614731 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:58.614698 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754510-vbcd5_cfec9c71-c116-4565-bd79-d018a784cc85/cleanup/0.log" Jul 28 20:57:59.668055 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.668025 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55_a9a5119b-f9d3-4818-91be-1acd26aa5cf0/util/0.log" Jul 28 20:57:59.676668 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.676642 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55_a9a5119b-f9d3-4818-91be-1acd26aa5cf0/pull/0.log" Jul 28 20:57:59.682930 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.682909 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55_a9a5119b-f9d3-4818-91be-1acd26aa5cf0/extract/0.log" Jul 28 20:57:59.799522 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.799490 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp_c518a04d-3e69-4cb7-bf5f-3d3566518e95/util/0.log" Jul 28 20:57:59.805366 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.805344 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp_c518a04d-3e69-4cb7-bf5f-3d3566518e95/pull/0.log" Jul 28 20:57:59.811535 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.811512 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp_c518a04d-3e69-4cb7-bf5f-3d3566518e95/extract/0.log" Jul 28 20:57:59.923754 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.923670 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx_cbdcb3bc-83fa-4b3e-8a37-17906f715a0a/pull/0.log" Jul 28 20:57:59.930489 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.930455 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx_cbdcb3bc-83fa-4b3e-8a37-17906f715a0a/extract/0.log" Jul 28 20:57:59.942177 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:57:59.942148 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx_cbdcb3bc-83fa-4b3e-8a37-17906f715a0a/util/0.log" Jul 28 20:58:00.069469 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:00.069441 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7_38ebac93-002a-4b92-8441-cd3916ea3d07/pull/0.log" Jul 28 20:58:00.075086 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:00.075066 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7_38ebac93-002a-4b92-8441-cd3916ea3d07/extract/0.log" Jul 28 20:58:00.083303 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:00.083281 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7_38ebac93-002a-4b92-8441-cd3916ea3d07/util/0.log" Jul 28 20:58:00.920266 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:00.920235 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tth4c_7c93be44-5de6-43f6-9ff0-2f9f7124fb7d/limitador/0.log" Jul 28 20:58:01.439005 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:01.438977 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cqn78d_b560ce9e-22dd-4f93-8492-ade45b36c42c/istio-proxy/0.log" Jul 28 20:58:01.559517 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:01.559482 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xckd2_da791d92-4c73-4d7c-85b4-ec422ff63696/discovery/0.log" Jul 28 20:58:12.385763 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:12.385731 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sq8gz_8f668389-c6ba-44b7-aded-ab82eb3cc122/global-pull-secret-syncer/0.log" Jul 28 20:58:12.529565 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:12.529536 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z4rd8_ee3319a3-c972-4ea8-9a22-7faa5baed072/konnectivity-agent/0.log" Jul 28 20:58:12.581440 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:12.581406 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-255.ec2.internal_cfbeafd1998bc253fc31c2c8fabaa146/haproxy/0.log" Jul 28 20:58:16.545555 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.545528 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55_a9a5119b-f9d3-4818-91be-1acd26aa5cf0/extract/0.log" Jul 28 20:58:16.569364 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.569340 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55_a9a5119b-f9d3-4818-91be-1acd26aa5cf0/util/0.log" Jul 28 20:58:16.593868 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.593840 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lzr55_a9a5119b-f9d3-4818-91be-1acd26aa5cf0/pull/0.log" Jul 28 20:58:16.623590 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.623568 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp_c518a04d-3e69-4cb7-bf5f-3d3566518e95/extract/0.log" Jul 28 20:58:16.650069 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.650045 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp_c518a04d-3e69-4cb7-bf5f-3d3566518e95/util/0.log" Jul 28 20:58:16.688640 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.688620 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mpmdp_c518a04d-3e69-4cb7-bf5f-3d3566518e95/pull/0.log" Jul 28 20:58:16.728932 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.728913 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx_cbdcb3bc-83fa-4b3e-8a37-17906f715a0a/extract/0.log" Jul 28 20:58:16.753642 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.753622 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx_cbdcb3bc-83fa-4b3e-8a37-17906f715a0a/util/0.log" Jul 28 20:58:16.778436 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.778414 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73z7kjx_cbdcb3bc-83fa-4b3e-8a37-17906f715a0a/pull/0.log" Jul 28 20:58:16.827435 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.827377 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7_38ebac93-002a-4b92-8441-cd3916ea3d07/extract/0.log" Jul 28 20:58:16.845909 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.845884 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7_38ebac93-002a-4b92-8441-cd3916ea3d07/util/0.log" Jul 28 20:58:16.872458 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:16.872437 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1dpjt7_38ebac93-002a-4b92-8441-cd3916ea3d07/pull/0.log" Jul 28 20:58:17.790081 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:17.790048 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tth4c_7c93be44-5de6-43f6-9ff0-2f9f7124fb7d/limitador/0.log" Jul 28 20:58:19.613113 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.613043 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce04cd2d-19af-411d-9316-50c8cb949b5a/alertmanager/0.log" Jul 28 20:58:19.637337 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.637309 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce04cd2d-19af-411d-9316-50c8cb949b5a/config-reloader/0.log" Jul 28 20:58:19.663561 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.663543 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce04cd2d-19af-411d-9316-50c8cb949b5a/kube-rbac-proxy-web/0.log" Jul 28 20:58:19.687966 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.687941 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce04cd2d-19af-411d-9316-50c8cb949b5a/kube-rbac-proxy/0.log" Jul 28 20:58:19.715588 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.715570 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce04cd2d-19af-411d-9316-50c8cb949b5a/kube-rbac-proxy-metric/0.log" Jul 28 20:58:19.741333 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.741308 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce04cd2d-19af-411d-9316-50c8cb949b5a/prom-label-proxy/0.log" Jul 28 20:58:19.766475 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.766452 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ce04cd2d-19af-411d-9316-50c8cb949b5a/init-config-reloader/0.log" Jul 28 20:58:19.857172 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.857146 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-v9k6d_591c6c2d-05f6-43a0-a8b9-4fbc05852c2f/kube-state-metrics/0.log" Jul 28 20:58:19.881899 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.881832 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-v9k6d_591c6c2d-05f6-43a0-a8b9-4fbc05852c2f/kube-rbac-proxy-main/0.log" Jul 28 20:58:19.908580 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.908554 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-v9k6d_591c6c2d-05f6-43a0-a8b9-4fbc05852c2f/kube-rbac-proxy-self/0.log" Jul 28 20:58:19.941010 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.940991 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bc6fdb7cf-v75qc_c62c6f2b-4184-4048-84f1-955575f8a2d9/metrics-server/0.log" Jul 28 20:58:19.969746 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:19.969718 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-fwvp7_a02098c4-4afc-461f-b380-3df3cc327c92/monitoring-plugin/0.log" Jul 28 20:58:20.091130 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.091096 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8kjr_2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0/node-exporter/0.log" Jul 28 20:58:20.116029 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.116001 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8kjr_2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0/kube-rbac-proxy/0.log" Jul 28 20:58:20.149900 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.149851 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c8kjr_2541d2b1-8fd7-4d4c-9dfa-e73f4b61f9a0/init-textfile/0.log" Jul 28 20:58:20.341537 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.341518 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5qckb_f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6/kube-rbac-proxy-main/0.log" Jul 28 20:58:20.366475 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.366450 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5qckb_f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6/kube-rbac-proxy-self/0.log" Jul 28 20:58:20.392037 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.392020 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5qckb_f3d3ebc7-9793-4df3-bfdf-1598eb7ee9d6/openshift-state-metrics/0.log" Jul 28 20:58:20.633013 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.632984 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-zvs4w_3cf84def-1c6b-4cd9-8d66-8798b4360452/prometheus-operator/0.log" Jul 28 20:58:20.659356 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.659272 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-zvs4w_3cf84def-1c6b-4cd9-8d66-8798b4360452/kube-rbac-proxy/0.log" Jul 28 20:58:20.689536 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.689514 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2v8l9_a2bcc05b-3fc3-4ae0-97a0-b382cb647ecb/prometheus-operator-admission-webhook/0.log" Jul 28 20:58:20.719704 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.719683 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dd8bb8f5-hh429_c586b467-e767-4d53-bc94-da07eac07192/telemeter-client/0.log" Jul 28 20:58:20.745520 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.745450 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dd8bb8f5-hh429_c586b467-e767-4d53-bc94-da07eac07192/reload/0.log" Jul 28 20:58:20.772765 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.772745 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dd8bb8f5-hh429_c586b467-e767-4d53-bc94-da07eac07192/kube-rbac-proxy/0.log" Jul 28 20:58:20.808461 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.808441 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54bbc586dd-qj699_c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78/thanos-query/0.log" Jul 28 20:58:20.831876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.831860 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54bbc586dd-qj699_c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78/kube-rbac-proxy-web/0.log" Jul 28 20:58:20.857996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.857975 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54bbc586dd-qj699_c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78/kube-rbac-proxy/0.log" Jul 28 20:58:20.887334 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.887306 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54bbc586dd-qj699_c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78/prom-label-proxy/0.log" Jul 28 20:58:20.912677 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.912656 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54bbc586dd-qj699_c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78/kube-rbac-proxy-rules/0.log" Jul 28 20:58:20.936234 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:20.936217 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-54bbc586dd-qj699_c3d6c98f-8cc3-42e7-a38d-08af6b4a9b78/kube-rbac-proxy-metrics/0.log" Jul 28 20:58:22.226354 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226311 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh"] Jul 28 20:58:22.226876 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226855 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78296855-fbc4-4910-ac32-e9deda856149" containerName="payload-pre-processing" Jul 28 20:58:22.226962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226879 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="78296855-fbc4-4910-ac32-e9deda856149" containerName="payload-pre-processing" Jul 28 20:58:22.226962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226891 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" Jul 28 20:58:22.226962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226899 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" Jul 28 20:58:22.226962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226920 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558c5b3c-542d-46ea-8b9d-bd3df13777bd" containerName="payload-pre-processing" Jul 28 20:58:22.226962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226929 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="558c5b3c-542d-46ea-8b9d-bd3df13777bd" containerName="payload-pre-processing" Jul 28 20:58:22.226962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226945 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" Jul 28 20:58:22.226962 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226953 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226966 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427171fc-9f77-46d6-a210-63f7b27504d3" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226976 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="427171fc-9f77-46d6-a210-63f7b27504d3" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226990 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.226999 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227012 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62265ad6-038a-4f50-8dc5-61ad7d929665" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227023 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="62265ad6-038a-4f50-8dc5-61ad7d929665" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227034 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227043 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227054 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7925da09-1920-463d-ab3c-4c48782cec9d" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227063 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7925da09-1920-463d-ab3c-4c48782cec9d" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227077 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc212fd-0985-4b92-8c75-8c6da67c7add" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227085 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc212fd-0985-4b92-8c75-8c6da67c7add" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227096 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4186e0f0-0d24-44bc-bf46-dbae729e37ae" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227104 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4186e0f0-0d24-44bc-bf46-dbae729e37ae" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227120 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef1f1de-b499-4926-af32-ffa76297d75e" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227129 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef1f1de-b499-4926-af32-ffa76297d75e" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227221 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="939eece6-bffd-4bc9-9ea1-e4ff8e9f4c36" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227234 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="62265ad6-038a-4f50-8dc5-61ad7d929665" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227245 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227254 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227263 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227272 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef1f1de-b499-4926-af32-ffa76297d75e" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227282 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4186e0f0-0d24-44bc-bf46-dbae729e37ae" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227291 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc212fd-0985-4b92-8c75-8c6da67c7add" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227301 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="78296855-fbc4-4910-ac32-e9deda856149" containerName="payload-pre-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227310 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227317 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19fdb3a-f626-4d8c-b7d9-daeefa9c0387" containerName="payload-processing" Jul 28 20:58:22.227339 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227353 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="558c5b3c-542d-46ea-8b9d-bd3df13777bd" containerName="payload-pre-processing" Jul 28 20:58:22.228712 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227361 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="427171fc-9f77-46d6-a210-63f7b27504d3" containerName="payload-pre-processing" Jul 28 20:58:22.228712 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.227369 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7925da09-1920-463d-ab3c-4c48782cec9d" containerName="payload-processing" Jul 28 20:58:22.231007 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.230989 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.233060 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.233038 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kdk6l\"/\"default-dockercfg-cqn65\"" Jul 28 20:58:22.233170 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.233080 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdk6l\"/\"openshift-service-ca.crt\"" Jul 28 20:58:22.233609 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.233592 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdk6l\"/\"kube-root-ca.crt\"" Jul 28 20:58:22.240751 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.240720 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh"] Jul 28 20:58:22.290178 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.290156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-lib-modules\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.290274 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.290209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2lz\" (UniqueName: \"kubernetes.io/projected/c9ca0013-8d2f-4414-827e-68bca73c5f2d-kube-api-access-nq2lz\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.290274 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.290257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-proc\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.290371 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.290288 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-sys\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.290411 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.290374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-podres\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2lz\" (UniqueName: \"kubernetes.io/projected/c9ca0013-8d2f-4414-827e-68bca73c5f2d-kube-api-access-nq2lz\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391748 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-proc\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391748 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391678 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-sys\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391748 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-podres\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391748 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391728 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-lib-modules\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391766 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-sys\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391766 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-proc\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391863 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-podres\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.391946 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.391880 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9ca0013-8d2f-4414-827e-68bca73c5f2d-lib-modules\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.399789 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.399769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2lz\" (UniqueName: \"kubernetes.io/projected/c9ca0013-8d2f-4414-827e-68bca73c5f2d-kube-api-access-nq2lz\") pod \"perf-node-gather-daemonset-82sgh\" (UID: \"c9ca0013-8d2f-4414-827e-68bca73c5f2d\") " pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.542007 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.541986 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:22.685100 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.678226 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh"] Jul 28 20:58:22.784560 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:22.784528 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" event={"ID":"c9ca0013-8d2f-4414-827e-68bca73c5f2d","Type":"ContainerStarted","Data":"308183d81bb681421681e8692f99302241e5e83ec66849d13bce500080dabc31"} Jul 28 20:58:23.354167 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:23.354140 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c4b669769-nxc5w_cc25ee02-d4ec-44b6-8262-44ab0cd0cc72/console/0.log" Jul 28 20:58:23.791005 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:23.790970 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" event={"ID":"c9ca0013-8d2f-4414-827e-68bca73c5f2d","Type":"ContainerStarted","Data":"88e3c96be65af338398adc8f2f90f2ef69453acc8de75addbb1374820023a6ce"} Jul 28 20:58:23.791180 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:23.791061 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:23.807446 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:23.807375 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" podStartSLOduration=1.807316098 podStartE2EDuration="1.807316098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:58:23.806991719 +0000 UTC m=+3531.251302736" watchObservedRunningTime="2026-07-28 20:58:23.807316098 +0000 UTC m=+3531.251627116" Jul 28 20:58:25.181151 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:25.181115 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-29jh7_17856a56-d18c-43ad-9049-0aad056adbd8/dns/0.log" Jul 28 20:58:25.204420 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:25.204400 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-29jh7_17856a56-d18c-43ad-9049-0aad056adbd8/kube-rbac-proxy/0.log" Jul 28 20:58:25.358338 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:25.358306 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7mnmq_96c649c0-feb1-43e1-98d5-c31848ac7b64/dns-node-resolver/0.log" Jul 28 20:58:26.009647 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:26.009617 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5vfv9_c4e9a8ae-a566-49ae-a090-aae3e55b73e9/node-ca/0.log" Jul 28 20:58:27.049342 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:27.049298 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cqn78d_b560ce9e-22dd-4f93-8492-ade45b36c42c/istio-proxy/0.log" Jul 28 20:58:27.209989 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:27.209959 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xckd2_da791d92-4c73-4d7c-85b4-ec422ff63696/discovery/0.log" Jul 28 20:58:28.339534 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:28.339501 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rgzjl_179eb78a-da87-466d-b86b-70002c68970a/serve-healthcheck-canary/0.log" Jul 28 20:58:29.022699 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:29.022672 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mcxfq_d99636cc-df6d-421b-96bf-1523e71f523f/kube-rbac-proxy/0.log" Jul 28 20:58:29.048101 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:29.048074 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mcxfq_d99636cc-df6d-421b-96bf-1523e71f523f/exporter/0.log" Jul 28 20:58:29.073049 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:29.073021 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mcxfq_d99636cc-df6d-421b-96bf-1523e71f523f/extractor/0.log" Jul 28 20:58:29.807082 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:29.807055 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kdk6l/perf-node-gather-daemonset-82sgh" Jul 28 20:58:32.286881 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:32.286853 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-5tjgw_3ca056c4-b45b-42e3-80e7-2483d9b2f0d6/manager/0.log" Jul 28 20:58:32.313488 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:32.313460 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685bc94585-5tjgw_3ca056c4-b45b-42e3-80e7-2483d9b2f0d6/copy-manifests/0.log" Jul 28 20:58:32.964199 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:32.964157 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cqn78d_b560ce9e-22dd-4f93-8492-ade45b36c42c/istio-proxy/0.log" Jul 28 20:58:33.105735 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:33.105699 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xckd2_da791d92-4c73-4d7c-85b4-ec422ff63696/discovery/0.log" Jul 28 20:58:33.644868 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:33.644836 2581 scope.go:117] "RemoveContainer" containerID="b32d352fee7138616e6af1ea0b54cc0950e59c687087da3b2338871aefcdab0d" Jul 28 20:58:33.652793 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:33.652768 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-789bdb4f85-6qxsj_941dff99-8dbf-4928-91c7-3baf76d0919b/manager/0.log" Jul 28 20:58:33.747662 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:33.747633 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fmhmm_61d2fa7c-e100-4e08-a5ba-c251999de893/openshift-lws-operator/0.log" Jul 28 20:58:34.647785 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:34.647754 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cqn78d_b560ce9e-22dd-4f93-8492-ade45b36c42c/istio-proxy/0.log" Jul 28 20:58:34.785974 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:34.785943 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-xckd2_da791d92-4c73-4d7c-85b4-ec422ff63696/discovery/0.log" Jul 28 20:58:40.564345 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.564301 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmjk4_b5b556cf-24a3-4fcf-a764-d2f748624105/kube-multus-additional-cni-plugins/0.log" Jul 28 20:58:40.587993 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.587973 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmjk4_b5b556cf-24a3-4fcf-a764-d2f748624105/egress-router-binary-copy/0.log" Jul 28 20:58:40.614213 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.614193 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmjk4_b5b556cf-24a3-4fcf-a764-d2f748624105/cni-plugins/0.log" Jul 28 20:58:40.637669 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.637643 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmjk4_b5b556cf-24a3-4fcf-a764-d2f748624105/bond-cni-plugin/0.log" Jul 28 20:58:40.665571 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.665554 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmjk4_b5b556cf-24a3-4fcf-a764-d2f748624105/routeoverride-cni/0.log" Jul 28 20:58:40.691291 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.691272 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmjk4_b5b556cf-24a3-4fcf-a764-d2f748624105/whereabouts-cni-bincopy/0.log" Jul 28 20:58:40.715612 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.715592 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nmjk4_b5b556cf-24a3-4fcf-a764-d2f748624105/whereabouts-cni/0.log" Jul 28 20:58:40.754834 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.754812 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v22pp_e9090d6e-910d-40e6-8fee-7efa6e20cbe2/kube-multus/0.log" Jul 28 20:58:40.914515 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.914444 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xzqcj_6bca7392-4c0d-4e95-9e7c-3eb68914d111/network-metrics-daemon/0.log" Jul 28 20:58:40.938750 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:40.938726 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xzqcj_6bca7392-4c0d-4e95-9e7c-3eb68914d111/kube-rbac-proxy/0.log" Jul 28 20:58:41.798493 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:41.798464 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/ovn-controller/0.log" Jul 28 20:58:41.816127 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:41.816102 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/ovn-acl-logging/0.log" Jul 28 20:58:41.840938 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:41.840911 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/kube-rbac-proxy-node/0.log" Jul 28 20:58:41.868423 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:41.868399 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 20:58:41.890814 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:41.890796 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/northd/0.log" Jul 28 20:58:41.914603 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:41.914585 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/nbdb/0.log" Jul 28 20:58:41.940008 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:41.939992 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/sbdb/0.log" Jul 28 20:58:42.002622 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:42.002575 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kk9q6_28be3705-4916-4862-bd49-84caadef8558/ovnkube-controller/0.log" Jul 28 20:58:43.758996 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:43.758966 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vk459_f60c1e92-7837-41b8-9bf1-0bebd43c48c7/network-check-target-container/0.log" Jul 28 20:58:44.867397 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:44.867369 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mss6g_ab6059fd-4193-477e-b5f1-7dc7da431cb2/iptables-alerter/0.log" Jul 28 20:58:45.608387 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:45.608357 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-knj5g_fdfc0ce9-10c1-489b-acbf-21e213c48a11/tuned/0.log" Jul 28 20:58:49.821867 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:49.821837 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-cn4cn_099719d8-3d25-418e-b055-a4a1363e2765/csi-driver/0.log" Jul 28 20:58:49.844126 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:49.844105 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-cn4cn_099719d8-3d25-418e-b055-a4a1363e2765/csi-node-driver-registrar/0.log" Jul 28 20:58:49.869650 ip-10-0-136-255 kubenswrapper[2581]: I0728 20:58:49.869631 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-cn4cn_099719d8-3d25-418e-b055-a4a1363e2765/csi-liveness-probe/0.log" Jul 28 21:00:00.148591 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.148555 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw"] Jul 28 21:00:00.149007 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.148937 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" Jul 28 21:00:00.149007 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.148949 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab774a9-c389-4b5e-ae95-3ed75b46f741" containerName="istio-proxy" Jul 28 21:00:00.149007 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.148967 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" Jul 28 21:00:00.149007 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.148972 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="51786bf7-54c3-43fd-8b40-b49fa5b2f280" containerName="istio-proxy" Jul 28 21:00:00.152030 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.152013 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw" Jul 28 21:00:00.155031 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.155000 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-982zh\"" Jul 28 21:00:00.169520 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.169496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw" Jul 28 21:00:00.171448 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.171423 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw"] Jul 28 21:00:00.301549 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.301426 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw"] Jul 28 21:00:00.303739 ip-10-0-136-255 kubenswrapper[2581]: W0728 21:00:00.303712 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f48c9e_5479_4581_ba45_025bafbb8a9d.slice/crio-4dda0816a7a897f1caca4d5eb9e479d8e93e1545031ac3f2b997af38b38e5fb2 WatchSource:0}: Error finding container 4dda0816a7a897f1caca4d5eb9e479d8e93e1545031ac3f2b997af38b38e5fb2: Status 404 returned error can't find the container with id 4dda0816a7a897f1caca4d5eb9e479d8e93e1545031ac3f2b997af38b38e5fb2 Jul 28 21:00:00.320766 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:00.320738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw" event={"ID":"39f48c9e-5479-4581-ba45-025bafbb8a9d","Type":"ContainerStarted","Data":"4dda0816a7a897f1caca4d5eb9e479d8e93e1545031ac3f2b997af38b38e5fb2"} Jul 28 21:00:01.327173 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:01.327144 2581 generic.go:356] "Generic (PLEG): container finished" podID="39f48c9e-5479-4581-ba45-025bafbb8a9d" containerID="065647146a49081512d8c3279d1a1177d7148e987e7817f67ef713a14deab200" exitCode=0 Jul 28 21:00:01.327610 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:01.327207 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw" event={"ID":"39f48c9e-5479-4581-ba45-025bafbb8a9d","Type":"ContainerDied","Data":"065647146a49081512d8c3279d1a1177d7148e987e7817f67ef713a14deab200"} Jul 28 21:00:02.467726 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:02.467701 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw" Jul 28 21:00:03.338586 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:03.338552 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw" Jul 28 21:00:03.338739 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:03.338593 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754540-zjrxw" event={"ID":"39f48c9e-5479-4581-ba45-025bafbb8a9d","Type":"ContainerDied","Data":"4dda0816a7a897f1caca4d5eb9e479d8e93e1545031ac3f2b997af38b38e5fb2"} Jul 28 21:00:03.338739 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:03.338645 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dda0816a7a897f1caca4d5eb9e479d8e93e1545031ac3f2b997af38b38e5fb2" Jul 28 21:00:03.527760 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:03.527734 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf"] Jul 28 21:00:03.529789 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:03.529770 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-4svqf"] Jul 28 21:00:05.152602 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:05.152568 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="552a8af6-9749-4128-bd82-83cacf51515a" path="/var/lib/kubelet/pods/552a8af6-9749-4128-bd82-83cacf51515a/volumes" Jul 28 21:00:33.685501 ip-10-0-136-255 kubenswrapper[2581]: I0728 21:00:33.685468 2581 scope.go:117] "RemoveContainer" containerID="5c84883be114fe4d7e27504518b2c9191a29369a3ac0450f67a2a0e76af47e97"